AutoCua

Automatic Computer Use Agent.
Mac, Windows, Linux, real phone, simulator, any browser.
A multi-agent harness engine.

pip install AutoCua

| | Windows | Linux | macOS | |:--|:--:|:--:|:--:| | Computer use — any desktop app, with real mouse and keyboard input | ✓ | ✓ | ✓ | | Browser skills — Chrome, Firefox, Edge, Brave and Opera | ✓ | ✓ | ✓ | | Shell use — a coding agent in a real terminal | ✓ | ✓ | ✓ | | Web use — your own Chrome, driven through the AutoCua extension | ✓ | ✓ | ✓ | | Mobile use — your iPhone, iPad or an iOS Simulator | — | — | ✓ |

_{Windows 10 and 11 · Linux with GNOME · macOS on Apple Silicon or Intel}

Full capabilities: agent_capabilities.md

Computer use

Any desktop app, driven with real mouse and keyboard input. Agent mode: computer use.

https://github.com/user-attachments/assets/e6c4b637-baeb-441f-8dfb-e550d88b4223

Web agent: fast mode

6.1 seconds for a flight search. Agent mode: web.

https://github.com/user-attachments/assets/8cc797d5-5556-40a9-bbfc-5d6aa8880fb8

Web agent: quality mode

Under a minute for three tasks in one run: an iPhone 18 Pro into the Apple bag, a case into the Amazon basket, and a Google Sheet of Meta's five-year financials. Agent mode: web.

https://github.com/user-attachments/assets/82bf6163-55c9-4cc4-95e7-9c6466f2a25d

iOS agent: iPhone use

A paired iPhone, driven over the USB cable. Agent mode: mobile use, iOS.

https://github.com/user-attachments/assets/3c22eb45-d939-4742-9053-22ecc75b416f

Pick a surface

One string decides what the agent drives. Everything else stays the same.

from AutoCua.agent_launcher import run_agent
run_agent(mode="web use", provider="openrouter", model="gemini-3.8-flash",
          task="Open wikipedia.org and summarise today's featured article.")

| mode | What it drives | Where it runs | |---|---|---| | "computer use" | The desktop. Clicks, typing, hotkeys, apps, AppleScript | macOS, Windows or Linux, auto detected | | "shell use" | A coding agent, straight into a real terminal | macOS, Windows or Linux | | "web use" | Your own Chrome through the AutoCua extension; headless runs on a debugging port | Any host with Chrome, installed for you when it is missing | | "mobile use, ios" | An iOS Simulator by default, or your paired iPhone or iPad | macOS host | | "mobile use, android" | Not available yet | |

On Linux, computer use needs a GNOME desktop (tested on Wayland), and computer use and shell use both need a few distro packages that pip cannot install. See Linux, from pip.

Why it holds up on real software

No pixel guessing. The model never sees a coordinate. It picks a numbered element from a view that fuses the operating system's accessibility tree with OCR, and a number it was never shown is rejected before anything moves.

No debugging port on your desktop. Input goes in at the operating system's own input layer. Mouse events posted at kCGHIDEventTap on macOS, pywinauto's real input path on Windows, and a kernel driver when Windows UAC blocks everything else. An ordinary application sees the same events it would get from your hand on the mouse.

No debugging port on your browser either. A normal "web use" run drives your own Chrome through the AutoCua extension, which Chrome loads at launch with no store install. Chrome shows its own "AutoCua started debugging this browser" bar while the agent works, and its Cancel button hands the browser back to you. A debugging port is opened only for headless runs, on a browser of the agent's own. The page scanner never calls Runtime.enable, and the numbered boxes are painted into the screenshot, not into your DOM.

No context wall. A separate compression agent watches the transcript and splices a handoff summary into it mid-run, so a long task finishes instead of dying at the context limit. In the app you watch the memory ring's count fall.

No telemetry. AutoCua sends nothing anywhere except to the model provider you chose, and to Telegram if you connect it yourself.

Contents

• Quick start
• How it sees a screen
• How it acts on a screen
• The agent team
• The browser agent
• iPhone and iPad
• Windows, UAC and the kernel driver
• Automation testing and workflows
• Running tasks in parallel
• The desktop app
• Memory, vault, chats and skills
• Remote control from Telegram
• Providers and models
• Requirements and setup
• Limits and safety
• Project layout
• Roadmap
• Licence, credits and citation

Quick start

pip install AutoCua

On Linux, install a few distro packages first: see Linux, from pip.

Put the four lines from Pick a surface in a main.py, then:

python main.py

That is the whole API.

What the wheel brings, and what it cannot

Two of the native pieces arrive on their own, and one, the desktop app's Electron, when you say yes to it. Two cannot: a Windows kernel driver, which pip is structurally unable to install, and on Linux the desktop's accessibility and input bindings, which come from your distro.

| | How it reaches you | |---|---| | Browser agent — Rust | Ships prebuilt in the Windows x64, Apple Silicon macOS and Linux x86_64 wheels. On any other platform pip builds it from the source distribution, which needs Rust and a C compiler. | | WebDriverAgent — iOS | Cloned automatically the first time you run a "mobile use, ios" task. Needs full Xcode and git. Nothing to do by hand. | | Electron — the desktop app's Chromium window | Downloaded only if you say yes. The first ui=True launch from a terminal asks (120 to 160 MB, once) on macOS 13+, Windows 10+ and Linux, except stock Ubuntu 23.10+ (below); without it the app opens in pywebview. python -m AutoCua.utils.electron fetches it any time. No Node.js needed. | | Interception — Windows UAC | Not installed by pip. One manual step, once — below. | | AT-SPI, GTK and the desktop portal — Linux | Not installed by pip. One apt line, once — below. |

Answering UAC prompts on Windows

Windows' user-mode SendInput cannot deliver input to the UAC secure desktop, so an elevation prompt can only be answered from kernel mode. That is the one thing the Interception driver does here.

pip install cannot set it up, and no packaging trick changes that: a wheel is unpacked and never executed, pip does not run elevated, and the driver binds its input slots only when it loads at boot. Without it every mode still works — you simply lose UAC handling, and a task that hits an elevation prompt stops there.

To enable it, pick one:

setup_windows.bat

from a checkout — it asks first (answer N and nothing is downloaded or installed), then downloads the pinned release, verifies its SHA-256, installs it, binds it to your built-in keyboard only so external keyboards are never filtered, and asks before rebooting. Or install Interception yourself from its own release page and reboot.

It is never bundled because it is dual-licensed LGPL-3.0 non-commercial / paid commercial.

Shipping it inside the wheel would make this project its distributor to every person who runs

pip install, which that licence does not allow for free. You receive it from its author.

See THIRD_PARTY_NOTICES.md.

Linux, from pip

The Linux wheel is built for x86_64 and needs a GNOME desktop (tested on Ubuntu with Wayland; X11 is not tested yet). The agent reads and drives the desktop through AT-SPI, GTK and the XDG RemoteDesktop portal. Those bindings come from your distro, not from pip, so install them once:

sudo apt install python3-gi python3-cairo python3-gi-cairo python3-venv gir1.2-atspi-2.0 gir1.2-gtk-3.0 \
                 gir1.2-webkit2-4.1 at-spi2-core xdg-desktop-portal xdg-desktop-portal-gnome \
                 gir1.2-gstreamer-1.0 gstreamer1.0-pipewire gstreamer1.0-plugins-base \
                 libglib2.0-bin libgtk-3-bin xdg-utils

On KDE, swap xdg-desktop-portal-gnome for xdg-desktop-portal-kde. Fedora, Arch and openSUSE ship the same packages under their own names; setup_linux.sh in the repository lists them.

Optional, for faster recordings: also install your distro's ffmpeg. Each run's recording is then compressed on the GPU instead of the CPU, because the ffmpeg pip brings on Linux has no GPU encoders:

sudo apt install ffmpeg            # Debian, Ubuntu
sudo pacman -S --needed ffmpeg     # Arch

Skip it on Fedora and openSUSE: their own ffmpeg builds leave out codecs the app needs (RPM Fusion and Packman have full ones). Without it, recordings are still compressed, on the CPU.

Ubuntu and Debian refuse pip install into the system Python, so create a virtualenv first, inside the folder that holds your main.py and .env so the agent finds that .env. Build it on your distro's own python3 (3.10 to 3.14): AutoCua loads the system PyGObject from inside the venv only for the same 3.x version, so a venv on another version stops with "cannot import gi".

cd my-agent                         # the folder with your main.py and .env
python3 -m venv .venv
source .venv/bin/activate
pip install AutoCua
python main.py                      # the main.py from Quick start

The browser agent is prebuilt in the wheel, so you need no Rust or C compiler. On the first computer-use run GNOME asks for screen-share consent: turn Allow Remote Interaction on before you click Share, because GNOME leaves it off. Your choice is saved in AutoCua_data/. The first scan also downloads the PP-OCRv6 models that read text the desktop does not publish.

A few things worth knowing on the first run

• Add ui=True to open the desktop app instead of running the task in the terminal. The first

• Put your provider key in a .env file, or paste it into Settings in the app.
• Model names come from model_list.txt and are case sensitive. A name

• Everything the agent keeps, chats, keys, your own skills and browser profiles, lands in an

From a checkout

git clone https://github.com/AutoCua/AutoCua.git
cd AutoCua
bash setup_mac.sh            # macOS
setup_windows.bat            # Windows, self-elevates, asks before installing the kernel driver
cp .env.example .env         # add your provider key
python main.py

main.py is the whole configuration surface: one run_agent(...) call. Edit it in place:

run_agent(
    mode="computer use",
    provider="anthropic",
    model="claude-sonnet-5",           # names come from model_list.txt
    task="check the version of macOS.",
)

Every optional flag goes straight into that call (ui=True, speed="fast", …). All of them, ui, save_conversation, speed, external_terminal, headless, browser_port, browser_profile, extra_tasks, device, ios_version, sim_device, app_build, api_key and os, are documented in agent_capabilities.md.

How it sees a screen

Every step the agent gets an element tree and a matching annotated image, then reasons over both.

macOS

1. Find the real front window. Window z-order comes from CGWindowListCopyWindowInfo, and

1. Wake the app up. AXEnhancedUserInterface is set when it is not already on, so apps

1. OCR whatever the tree missed. Apple Vision runs at the accurate level with a confidence

1. Drop what cannot be clicked. Every element is re-tested against the real window stack

1. Draw the numbers. The capture is downscaled first to fit inside full HD (1920 x 1080,

Windows

The same contract, built out of what Windows offers.

One UI Automation cached round trip pulls 18 properties for the whole subtree, then four threads run in parallel: WinRT OCR, a taskbar walk, a Win32 backend scan and a raw UIA scan of the Start menu. The results are merged and deduped at 5 px, with OCR lines nested into the deepest container that does not already hold a labelled element there. Chromium browsers are launched with --force-renderer-accessibility so they publish their full tree, and page loads are gated by polling the toolbar reload button until it stops saying "Stop".

Clicks aim at the pixel centroid of the element's own content rather than the middle of its box, so a wide button with a label on the left still gets hit where the label is.

Both platforms

A step that only digests a tool result, the turn after a web lookup or after the coder agent reports back, deliberately skips the scan entirely. No screenshot, no tree, just the payload and an instruction to pull the findings into the scratchpad. Scanning a screen that nobody looked at costs tokens and buys nothing.

How it acts on a screen

Nothing in the desktop input path opens a debugging port, sets an automation flag or injects JavaScript. Input goes in at the operating system's own input layer, so an ordinary application sees the same events it would get from your hand on the mouse.

macOS

Mouse events are posted at kCGHIDEventTap, the same tap a physical mouse feeds, from a private event source, after a genuine cursor warp with CGWarpMouseCursorPosition.

• A partially visible element tries the accessibility AXPress action first, then falls back

• Multi-line text is put on the clipboard and pasted, then the previous clipboard is restored,

• Drags are 20 interpolated move events at 15 ms, not a teleport.
• applescript is a first-class tool, not a bash escape hatch. The model supplies a complete

Typing and hotkeys go through pyautogui and pynput rather than the private event source, so they are ordinary synthetic input.

Windows

Element clicks use pywinauto.click_input(), which is a real SendInput. When that fails because User Interface Privilege Isolation blocked it, the click is retried through the Interception kernel-mode driver. Typing switches to the driver when the front application is Windows Security. See Windows, UAC and the kernel driver.

A watcher for permission dialogs

macOS asks for consent the first time an app does something new, and a modal dialog will deadlock a run. AutoCua fingerprints those dialogs structurally, any window or sheet carrying a "Don't Allow" or "Deny" button, and clicks the affirmative. It runs on a one second loop around every AppleScript call and every shell command, and if a dialog is present but cannot be clicked, the command fails after 8 seconds with a permission-specific error instead of hanging forever.

Browser windows on the desktop

When a browser is in front, the agent gets browser rules injected into its prompt and the runtime changes behaviour:

• It reuses the browser you already have open, with your profile, cookies and sessions. The

• macOS waits for a real AXWebArea before scanning, with a 1.5 second grace period and a 15

• Known destinations are opened in one command with the query already in the URL, rather than

Tools the desktop agent has

| | macOS | Windows | |---|---|---| | Click | left_click (1, 2 or 3 clicks), right_click | same | | Type | input, typewrite | same | | Navigate | scroll, hotkey, screenshot, drag_drop | scroll, hotkey, screenshot | | System | open_app, shell (zsh), applescript | open_app, shell (PowerShell) | | Delegate | sub_agent (coder_agent, browser_agent), agent_wait, web | same | | State | todo_list, update_todo, scratchpad, wait, done | same |

19 tools on macOS, 17 on Windows. The names are a frozen allow-list. Anything else the model invents comes back as an error result and is never executed.

The agent team

Behind one task sits a hierarchy of agents, each in its own process.

The parent agent owns the screen and decides what a task needs. When it needs real code work it delegates and can carry on, or block until the result comes back.

The coder agent has 14 tools: shell, view, grep, glob, write, replace, web, plan, todo_list, update_todo, wait, scratchpad, minion and exit. Its operating procedure is EXPLORE, PLAN, EXECUTE, VERIFY, and it is written into the prompt with teeth:

• Exploration is delegated by rule. "Never read the codebase first-hand to build first-time

• The plan is a structured Markdown document with real path:line anchors, not a restatement

• Every change with logic in it needs a throwaway test under ./.AutoCua_verify/ that is

• replace verifies the old block before writing and re-verifies after, so a stale line number

Minions are read-only scouts with six tools and no recursion. The restriction on tool names is structural: the minion's six-name set is the allow-list at the call router, so a hallucinated write comes back as an error result and never runs. Several minions fire from one coder step, run in parallel, and the loop blocks until all of them report back with findings anchored to exact path:line. Even a crashed minion returns its partial scratchpad, so the parent never hangs. To be precise about the boundary: the tool names are enforced, and the "read-only commands only" rule on its shell tool is held by the prompt.

Minions exist to keep the coder's context small. Every tool result is budgeted too: 200 lines or 15,000 characters, with shell keeping a 50-line tail, view capped at 2,000 lines and grep at 8 MB, so one wide search or noisy build log can never blow up the conversation.

Run either directly. They start in their own workspace, so point them at an absolute path when the work is in an existing project:

python -m AutoCua.mac.agent.coder   --task "refactor the auth module in /Users/me/projects/api" --provider anthropic --model claude-sonnet-5
python -m AutoCua.mac.agent.minions --task "where is _validate_token defined in /Users/me/projects/api and who calls it?" --provider anthropic --model claude-sonnet-5

The browser agent

"web use" is a different program from the rest of the framework. The whole browser agent, the agent loop, the page scanner and the controller, is one Rust crate compiled into a single Python extension module and imported like any other. Its LLM calls go through the same Python providers every other mode uses, in AutoCua/llm_provider/.

A checkout builds it on first import with cargo build --release and rebuilds only when a .rs file actually changed. The pip wheel ships it already compiled. One binary covers every CPython from 3.10 up.

Two ways into Chrome. Every run with a window goes through the AutoCua extension, a parallel run included: Chrome is started with the flags that load it, or taken as it is when it is already running with it, and the agents drive your own browser, logged in as you are, with no debugging port open; each agent is on a line of its own to the extension. A headless run uses the DevTools Protocol on a Chrome of the agent's own. Every tool works the same way in both. When Chrome is missing it is downloaded from Google and installed on the first web run.

Raw CDP, hand written

No Playwright, no Selenium, no Puppeteer, no chromedriver. None of them appear anywhere in the repository. In the debugging-port mode there is one WebSocket to Chrome plus hand-written HTTP for tab management, one session per tab and a child session per cross-origin iframe, attached in flatten mode up to six levels deep. In extension mode the same scanner runs as a content script of the extension, in an isolated world the page cannot see, and the tools send their input through chrome.debugger as the same trusted CDP events.

The property that matters:

**The debugging-port scanner runs no JavaScript in the page, and Runtime.enable is never

called in either mode.**

In the debugging-port mode the page model is two CDP calls per frame, DOMSnapshot.captureSnapshot plus Accessibility.getFullAXTree; in extension mode it is one injection into every frame of the tab, cross-origin frames included, reading the same facts from the DOM. Clicks and keystrokes go through CDP's trusted input path with a rising click count. Even the numbered boxes the model sees are painted into the JPEG in Rust with a hand-rolled 3x5 bitmap font, never into the DOM.

Two honest exceptions, both deliberate:

• run_script and scrape are the two doors for JavaScript in the page, and they are split by

• A scrape reads the screen the page is on plus the next two below it: three viewports at

• The "agent is driving" overlay, an edge glow and the agent's own cursor, is injected into the

The page model

• Settling counts in-flight requests rather than sleeping. 150 ms of quiet, capped at 3

• Occlusion by paint order demotes elements behind cookie banners and modals.
• Noise filtering collapses the wrappers and SVGs inside a button that would otherwise

• [1] is always the page itself. Numbering follows document order and stops at 300 elements.
• Per-host overrides are shipped, for example YouTube turns off pointer-cursor detection

• Modal dialogs are shown to the model as a step of their own, with their text, for it to

Staying logged in

A run with a window is in your own Chrome, so the agent is logged in wherever you are. Headless runs use a persistent profile keyed by name, so cookies, localStorage and logins survive between runs there too. As the source puts it, arriving already logged in matters far more than any fingerprint tuning, because a login wall is where a web agent usually stops.

18 tools: new_tab, switch_tab, close_tab, update_tab, navigate_tab, click, hold_click, input, keyboard, run_script, scroll, wait, scratchpad, todo_list, update_todo, skills, index_read, done, plus dialog on a step a popup holds. Scraping mode sends exactly six: index, more, exit_scrape_mode, scratchpad, update_todo, skills. A parallel agent pinned to one tab gets 15, with the three tab-lifecycle tools removed from its registry and refused at the router as a backstop.

iPhone and iPad

The same loop that drives your Mac drives a phone. Read the accessibility tree, annotate a screenshot, tap, swipe and type through WebDriverAgent. The model is never told which target it is on, because both paths end at the same endpoints.

| | device="simulation" (default) | device="hardware" | |---|---|---| | Runs on | An iOS Simulator on your Mac | Your paired iPhone or iPad | | Driven by | xcrun simctl plus one unsigned xcodebuild test | pymobiledevice3, USB forward plus XCUITest launch | | Needs | Full Xcode. No Apple ID, no signing, no pairing | One-time pairing and Team ID signing | | iOS version | ios_version="26.5", or the newest your Xcode can build for | Whatever the device runs | | Your own app build | app_build="MyApp-Simulator.ipa", installed before the task starts | Not supported, the run stops with an error | | Best for | Everyday runs, testing, pinning a specific iOS version | Real apps with your logins, camera, cellular |

run_agent(
    mode="mobile use, ios",
    provider="anthropic", model="claude-sonnet-5",
    device="simulation",        # or "hardware"
    ios_version=None,           # e.g. "26.5"; None picks the newest usable runtime
    sim_device="iphone",        # "iphone", "ipad", or an exact simulator name
    app_build=None,             # simulation only: a .ipa/.app in AutoCua_data/app_build/
    task="Open Settings, turn on Dark Mode, and confirm it on the home screen.",
)

AutoCua boots the simulator, starts WebDriverAgent, runs the task, then shuts everything down when the agent finishes, whether it succeeded, errored or you pressed Ctrl+C. The first run compiles WebDriverAgent once. After that it starts in well under a minute.

Testing your own app. app_build works with device="simulation" only. Put a simulator build (.ipa or .app) in AutoCua_data/app_build/ and pass its file name. It is installed the moment the simulator has booted, at the same time as WebDriverAgent starts, and the agent only begins once both are done. No app, no run: a missing file, a failed install or device="hardware" stops the run with a clear error before the agent starts. Details in agent_capabilities.md.

Three details worth knowing.

What simctl can boot is not the same as what xcodebuild can build for. Simulator runtimes are system wide and outlive Xcode upgrades, so AutoCua probes xcodebuild -showdestinations up front and filters to what is actually usable, instead of dying minutes into a run. If a parallel run needs one more device than you have, it creates one.

The element scan asks WebDriverAgent to skip its visible and accessible attributes, which it computes by hit-testing every element. On a 364-element home screen that takes the request from 3.60 s to 0.39 s, about 82 percent of the time. Visibility is recomputed from the element's centre, which is the exact point the tap will land.

There is a video_player tool because DRM players black out screenshots. When vision goes dark it drives play, pause, close and streaming checks from the accessibility tree, and proves "streaming" by watching the progress label advance across a four second window.

13 tools: open_app, click, input, scroll, wait, shell, web, vault, video_player, todo_list, update_todo, scratchpad, done.

Windows, UAC and the kernel driver

Windows UAC runs on a separate secure desktop. User-mode SendInput cannot reach it and screen capture fails there too. Most automation tools simply hang at an elevation prompt.

AutoCua turns that failure into the detector, then answers from kernel space.

1. Detect. The full-screen grab fails, so the scanner reports that UAC is up.
2. Ask the model. That step ships with no screenshot and no element tree. Just a

1. Inject. Those two combinations are intercepted and sent as raw scancodes through the

The driver is also the fallback whenever User Interface Privilege Isolation blocks ordinary input, for example when driving Windows Security.

How it is installed. setup_windows.bat asks first — answer N and nothing is downloaded or installed. On Y it downloads the author's own signed v1.0.1 release, verifies it against a pinned SHA-256 and aborts rather than run an unverified kernel installer. It then binds the driver to the built-in keyboard and mouse only, at the device level. That detail matters. The driver has ten keyboard slots that are never freed, so the older class-wide binding burned one on every wireless-keyboard reconnect and eventually left keyboards dead until reboot. Device-level binding takes exactly one slot at boot and never filters another keyboard. If the bind fails, setup strips every filter rather than reboot into a configuration that could kill your keyboard.

Without the driver everything still works except three things: answering a UAC prompt, a click that User Interface Privilege Isolation blocked, and typing into Windows Security.

Licensing. Interception is dual licensed. LGPL v3.0 for non-commercial use, and

commercial use requires a separate paid licence from its author. AutoCua's MIT licence

does not and cannot grant it. See THIRD_PARTY_NOTICES.md.

macOS has no UAC and AutoCua does not elevate on it. The analogue is TCC, handled entirely in user space by the permission wizard and the consent-dialog watcher. There is no sudo anywhere in the agent paths.

Automation testing and workflows

A task here is a sentence, not a selector. Nothing in it names an XPath, a CSS class or a coordinate, so the redesign that breaks a selector-based suite does not break the task. The agent finds the button again because it reads the screen the way a person does.

What that buys you.

| Need | How AutoCua covers it | |---|---| | End-to-end across surfaces | One harness covers a desktop app, a website and an iPhone. Change one string to move between them | | A device or browser matrix | extra_tasks=[...] runs N tasks at once. One Chrome with a tab each, or one simulator each | | Unattended runs | headless=True for Chrome. Desktop and iOS runs still need a logged-in graphical session | | An audit trail | save_conversation=True writes exactly what the model saw and answered at every step | | A result you can branch on | run_agent returns a status dict, and every parallel child writes its own result.json | | Credentials in a test | On iOS the vault types the secret. The value never enters the model's context | | Site or app specific rules | Skills are Markdown files loaded only when that site or app is in front | | Long flows | Memory compression keeps a long checkout or onboarding flow inside the context window |

A smoke test you can run today.

from AutoCua.agent_launcher import run_agent
result = run_agent(
    mode="web use", provider="anthropic", model="claude-sonnet-5", headless=True,
    save_conversation=True,
    task="""
    Go to staging.example.com, sign in as [email protected], add any item to the basket,
    go to checkout and stop at the payment step. Report the basket total and whether the
    payment form rendered. Do not submit a payment.
    """,
)
assert result["status"] == "success", result["message"]

run_agent returns {"status", "message"}, where status is success when the agent called done, error when the loop died, and incomplete when it was stopped or ran out of steps. That is the hook you branch on.

What you get back to read afterwards. With save_conversation=True, conversation/ holds the exact payload sent to the model at every step and raw_reasoning/ holds its raw reply, so a failed run is readable rather than guessed at. A parallel run additionally writes result.json per child and exits 0 when it returned, 1 on a crash and 130 on Ctrl+C.

Those folders are cleared at the start of every run, whether the flag is on or not. Copy out

anything you need to keep before running again.

One design decision matters more than any of this for reliability: the agent is told that a tool result reports that a tool ran, never that the screen changed. Every step judges the previous step's expected outcome against the new screenshot before it does anything else.

A cross-surface matrix in one call.

run_agent(
    mode="web use", provider="openrouter", model="gemini-3.8-flash", headless=True,
    task="Sign up for a new account on staging.example.com and confirm the welcome screen.",
    extra_tasks=[
        "Reset the password for [email protected] and confirm the email prompt appears.",
        "Open the pricing page and check every plan card shows a price and a CTA button.",
    ],
)

Each task gets its own tab, its own agent and its own working directory under ./parallel/task_N/, with a result.json holding status and message. Live output is prefixed per task, and one Ctrl+C stops all of them.

Everyday workflow automation is the same machinery pointed at your own machine:

task = "Open Numbers, put last month's totals from ~/Downloads/report.csv into a new sheet, chart them, and save it to the Desktop as monthly.numbers"
task = "Go through my Gmail inbox, find every invoice from this month, and save the PDFs into ~/Documents/Invoices"
task = "Check the three competitor pricing pages in my bookmarks and tell me what changed since the notes in ~/Desktop/pricing.md"

Be honest about what this is

AutoCua is a language model driving real software. Two runs of the same task can take different routes. It is a strong fit for exploratory testing, smoke flows, repetitive back-office work and anything where writing selectors costs more than the test is worth. It is not a replacement for a deterministic unit or integration suite, and it has no assertion framework of its own. Use it where a human tester would otherwise be clicking.

Running tasks in parallel

Pass extra_tasks and every task, including the first, runs at the same time in its own child process.

run_agent(
    mode="web use", provider="anthropic", model="claude-sonnet-5", headless=True,
    task="find the cheapest flight to Tokyo next month",
    extra_tasks=[
        "summarise today's top Hacker News thread",
        "check my GitHub notifications",
    ],
)

| | "web use" | "mobile use, ios" with device="simulation" | |---|---|---| | Each task gets | Its own agent pinned to its own tab | Its own simulator and its own WebDriverAgent port | | Shared | Your own Chrome, through the extension (headless: one browser of the agent's own on a debugging port); its front tab shows a cover page while the agents work in tabs behind it | Nothing. A phone screen cannot be split | | Isolation | A 13-tool single-tab registry. It cannot touch another agent's tab | Own port (8100, 8101, ...), own scratchpad, own build directory | | Ceiling | Tabs and RAM | Simulators, and it creates one when it needs another |

Backgrounded tabs are told they are still focused, so pages that gate on focus keep behaving while another agent is in front. Output is prefixed per task, one Ctrl+C stops everything, and results land in ./parallel/task_N/. On iOS every simulator the run booted is shut down at the end, whether it succeeded, failed or was interrupted. With app_build, every simulator installs your app at the same time, each as soon as it has booted.

./parallel/ is deleted and recreated on every parallel run. Copy out anything you want to

keep.

Parallel mode is for run_agent. The desktop app runs one agent at a time.

The desktop app

ui=True opens a native window, 1140 by 700, on a Flask server at AutoCua's own port (27321, or a free one for that run when another program holds it). The window is Chromium, through Electron, the same on every OS. Electron is not in the wheel, and nothing is downloaded without asking: on macOS 13 or newer, Windows 10 or newer, and 64-bit Linux, the first launch from a terminal asks whether to fetch it, and says how big it is and where it goes (a 120 to 160 MB download from Electron's own GitHub releases, the same zip npm install electron fetches, checked against a pinned SHA-256, and 300 to 390 MB on disk). y puts it in a per-user cache (~/Library/Caches/AutoCua/electron, %LOCALAPPDATA%\AutoCua\electron, ~/.cache/AutoCua/electron); no Node.js is needed. n is remembered and the question is not asked again; Enter, Ctrl+C, or no answer within two minutes means not now. With no terminal (an IDE's run button) or a hidden one, nothing is asked. python -m AutoCua.utils.electron fetches it any time (--force fetches it again). Deleting the cache folder removes it. ELECTRON_MIRROR and ELECTRON_CUSTOM_DIR fetch from a mirror, as they do for npm, and the question names it. A checkout that ran npm install in AutoCua/desktop (Node.js 22.12 or newer) uses that copy and is never asked.

Without Electron, or with AUTOCUA_UI=webview, the app opens in pywebview, with the OS title bar tinted to match the page so the whole surface reads as one. So does an Electron that cannot run here (macOS older than 13) or that fails before the app is up (a missing system library, a policy that blocks it). While the Chromium window is open, a second launch just brings it to the front.

On Linux, Chromium's sandbox needs a user namespace it can use, or its setuid helper, chrome-sandbox next to the executable, owned by root. Ubuntu 24.04 and newer (23.10 as an opt-in) take every capability away in the user namespaces of a program with no AppArmor profile (kernel.apparmor_restrict_unprivileged_userns=1), and a download made as you cannot make the helper root's, so there the Chromium window starts without the sandbox (--no-sandbox, for this window only) and the launch says so. The pywebview window has no sandbox there either: WebKitGTK's is off in it. To turn the sandbox on, make the helper root's, as Chrome's and Cursor's packages do, with the two commands the launch prints:

sudo chown root:root ~/.cache/AutoCua/electron/v<version>-linux-<arch>/chrome-sandbox
sudo chmod 4755 ~/.cache/AutoCua/electron/v<version>-linux-<arch>/chrome-sandbox

An update that brings a new Electron needs them again. An AppArmor profile that allows user namespaces for this Electron is not the way: it runs any app folder it is given, so the profile would let every program on the machine around Ubuntu's restriction, and everything the agent runs would inherit it.

+--------------+----------------------------------------------+
|  AutoCua     |  live agent screenshot  |  tracking progress |
|              |  (what the agent sees)  |  (scratchpad notes)|
|  + New chat  +-------------------------+--------------------+
|              |  tool-response chain    |  live TODO list    |
|  chat 1      |  "N tools used"         |  (agent's plan)    |
|  chat 2      +-------------------------+--------------------+
|  chat 3      |   Agent Notes  or  Skills   (big centre)     |
|              +----------------------------------------------+
|  settings    |   composer  [fast] [mode] [model] [skills]   |
+--------------+----------------------------------------------+

| Element | What it does | |---|---| | Live screenshot | The screen the agent just captured, updated every step | | Tracking progress | Each scratchpad entry streams in as a bullet on a connecting line | | Tool-response chain | Every tool call drawn as an animated canvas icon, so you can see why it did something | | Live TODO | The agent's own plan, tailed from its file as it edits it, frozen with crosses if you stop it | | Agent Notes | The final write-up, shown when a run ends, completed or stopped | | Skills | Browse, preview, add, edit and delete domain knowledge files, live, for desktop and iOS | | Memory ring | Context gauge beside the Skills icon, with the live token count inside. It breathes while compression runs and its count falls when the handoff lands | | Fast / Quality | Leaner prompt and fewer tokens per step, or full reasoning, which is the default | | Mode picker | Computer use, Mobile use, Shell use | | Stop | Retires the run id instantly, so a still-running model call can never repaint your next chat |

Chats are saved and resumable. Reopening one restores its history and puts the memory ring back where it was, and you can download the exact payload the model received. On macOS the first launch opens a setup wizard that walks the four permissions AutoCua needs, Accessibility, Full Disk Access, Screen Recording and Automation, one at a time, auto-advancing as each is granted. It reads the TCC database directly to notice a grant the in-process check cannot see, and repairs stale grants left by a previous build.

Two owners, one terminal prompt

In Shell use the terminal card has a single > prompt that either the agent or you can own.

| | AI mode | Manual mode | |---|---|---| | Header | AutoCua Code | AutoCua Terminal | | Prompt | Bare >, read only | Shows the real working directory | | Who runs it | The coder agent, spawning minions and writing files | You. No agent, no model | | Interrupt | The stop control | Ctrl+C, wired to a real signal on the process group |

Click the card and you take the keyboard. Type ls, git status, npm test and it runs live, streaming as it goes, with cd tracked between commands.

The bridge is the interesting part. Every command you run by hand is captured, command, directory, exit code and output, and replayed into the agent's next run as a block that tells it those effects are already applied. So you can drop in, check something yourself, fix a file, install a package, and the agent picks up already knowing. It is capped at 12 commands and 60 output lines each so it never bloats the conversation, and undelivered commands are re-queued if a run dies before it sees them.

Memory, vault, chats and skills

Memory compression is its own agent. When the live context crosses 110,000 tokens, a background thread asks a second model to write a handoff document, and the controller splices it into the transcript in place, on the main thread, guarded by a generation counter so a stale result can never land and a re-arm delay so it cannot thrash. Every agent uses it: the macOS and Windows desktop agents, iOS, the coder, and the Rust browser agent, which calls the same Python controller across the language boundary. It is why the memory ring's count in the app falls mid-run instead of only climbing.

The vault fills credentials without showing them to the model. The agent says "fill element 12 with the password". The runtime resolves the app from the element tree, looks up the credential locally, types it, and returns only "Credential filled successfully". The value never enters the model's context on the way in. Two limits worth stating: the store is plain JSON on your disk, and a field that renders its value back to the screen, a username rather than a masked password, will appear in the next element scan like any other text. PIN pads with a separate key per digit, like the iPhone lock screen, work the same way: the runtime taps the keys, checks that every press registered, and never logs which keys it pressed. It is wired into the iOS agent today.

Chats live outside the install folder in AutoCua_data/, so an uninstall cannot take them. They resume with the agent's full per-step reasoning paired to its tool results, plus a note recording how the previous run ended, on every path including stop and crash. Resumable chats are a feature of the desktop app. A terminal run with save_conversation=True writes readable per-step logs instead, which is what you want for auditing rather than resuming.

Skills are Markdown files matched to what is on screen, on the desktop and iOS agents. The browser agent has the slot but does not fill it yet. A router maps hostnames and app names to files, so opening Go

... (README truncated for length)