Profile
Back to NewsBack
GitHub Trending 7 min
Reader Mode
HKUDS/PhoneGo: PhoneGo: Personal AI Agent on Your Phone

HKUDS/PhoneGo: PhoneGo: Personal AI Agent on Your Phone

English · Chinese

PhoneGo: Personal AI Agent on Your Phone

CLI+GUI · Fast · Powerful · Reliable · Cost-Effective Phone-Use Agent

Download PhoneGo 0.4.0 for Android

Download PhoneGo and run it directly on your Android Phone.

Tell PhoneGo what you want. It opens and operates the right apps, gets the task done, and reports back.

  • Download and run directly on your phone.
  • Describe the task in natural language.
  • Let PhoneGo operate the relevant apps.
  • PhoneGo works in the background, so your phone remains yours to use.
  • Get the result when the task is complete.

What Makes PhoneGo Different?

⚡ Reusable App Commands

PhoneGo turns familiar app workflows into reusable commands:

  • Open a specific page.
  • Search for information.
  • Complete a booking or other multi-step task.
  • Replay tested workflows without rediscovering every tap.
Result: faster execution, more consistent outcomes, and fewer model calls.

📱 Visual App Interaction

For apps that require direct screen interaction, PhoneGo can:

  • Understand what is displayed on the screen.
  • Tap buttons and navigate menus.
  • Scroll through pages and return to previous screens.
  • Handle unfamiliar interfaces and unexpected app states.
Result: PhoneGo can work with apps even when no API or reusable command is available.

🔀 CLI (Command) + GUI (Visual) Execution

PhoneGo automatically chooses the most suitable approach:

  • Reusable commands for familiar, repeatable workflows.
  • Visual interaction when tapping or scrolling is required.
  • Vision models only for unfamiliar screens or new situations.
  • Seamless switching between commands and visual operations within the same task.
Result: command-level speed and reliability, with the flexibility of human-like app interaction.

🎬 Use Case

you   find a hotel near HKU on Xiaohongshu, pick one, then check tonight's price on Ctrip

app · Xiaohongshu: search → open a note → "Hopewell Hotel" · Ctrip: search Hopewell Hotel → 17–18 Nov · tonight: "no rooms for the dates you picked". Next available — 18–19 Nov from CNY 3379 · 19–20 Nov from CNY 3790 · 20 Nov from CNY 2303 · want me to move this hotel page to your phone to continue?

PhoneGo typing a hotel name it read on Xiaohongshu into Ctrip, then reporting tonight's prices


✨ What makes it different

| | | |---|---| | 🧭 It calls commands, it doesn't guess | Navigation is compiled offline into a map: one deterministic replay, no vision calls, a replayed tap in well under a second. | | 👁 It looks only when the map runs out | An unfamiliar screen goes to the vision model for that part alone — then you get a plain answer, not a log. | | 🔗 It carries what it read across apps | What it reads in one app it uses in the next: "see which hotel Xiaohongshu recommends, then check tonight's price on Ctrip". | | 📵 It doesn't take the phone | The app is driven on a screen of its own, invisible to you. Some apps only run on the main screen; PhoneGo follows them there, finishes, and puts them back. | | ↩️ It gives the phone back | When the task ends the app returns to its own screen, the chat comes forward, and it asks whether you want that page moved over to continue. | | 🔒 It stays on your device | Pairing keys, maps, sessions and settings live in the app. Only the screens it actually has to look at are sent. |

PhoneCLI's harness driving an Android emulator through the AndroidLab benchmark


⚙️ From app screens to commands

offline    a crawler walks an app and compiles it into a map   (once per app)
online     route the task -> replay the map's macro            (one small routing call)
fallback   unknown screen -> vision model drives that part     (only there)

A map is compiled once per app: every screen becomes a node, every element that leads somewhere an edge, and a task turns into an ordered list of steps. Running that list is the fast path — 64 s instead of 146 s on the same instruction — and the vision model stays the exception handler, not the engine.

📊 Measured on a real phone

| Task | Map + replay | Vision only | |---|---|---| | Find how to book a HKU visit (Xiaohongshu) | 17 steps / 64 s / 92.3k tokens | 32 steps / 146 s / 207.1k tokens |

Same phone, same instruction, both successful. The recordings live in docs/showcase/.

📲 Install

  1. Download PhoneGo-0.4.0-arm64.apk
(Android 14+, arm64) and open it on the phone. Allow installation from this source; MIUI asks once (tap "continue").
  1. Turn on wireless debugging, then enable PhoneGo's accessibility service:
the app reads the pairing code off the system dialog and pairs itself with the phone's own adbd, so the service has to be on first. No computer needed.
  1. Tap Settings → Model settings: pick a preset (OpenRouter, DeepSeek, Zhipu,
Alibaba Bailian, SiliconFlow, or an endpoint of your own), paste **your own API key, press Test connection**. Nothing is bundled.
  1. Type an instruction. Start with something that needs no login —
"open Settings and tell me which Wi-Fi is connected".

The interface is English or Chinese — it follows the phone's language, and Settings → Language overrides it. The walkthrough in docs/getting-started.md names every label both ways.

Updates install in place: every release is signed with the same key. The full walkthrough, including what each screen should look like and the five things that usually go wrong, is in docs/getting-started.md.

❓ Questions people ask

Does it upload my screen? Only the screens a model call has to look at, to the endpoint you configured. Maps, sessions, pairing keys and settings never leave the app.

Do I need a computer? No. The app uses the phone's own wireless-debugging connection; the same code also runs from a laptop over USB for development.

Which apps work? Any app you can drive by tapping. An app with a map is several times faster on the same task; the rest go through the vision fallback, which is slower and costs model calls. Maps are built per app, on your phone, and can be rebuilt any time.

Will it buy things for me? No. Anything that spends money (pay, order, submit) stops and asks first, and password or verification-code screens are handed to you.

📚 Documentation

| | | |---|---| | docs/manual.md | the manual: every screen, every setting, every failure | | docs/getting-started.md | from the APK to a finished task, with the first-run checks | | docs/product-app.md | what the app is, layer by layer | | docs/map-building.md | how a map is compiled, with the measurement behind each decision | | android-app/README.md | build it: Gradle, Chaquopy, the virtual-display helper | | docs/showcase/ | recordings, including the side-by-side comparison |

🌟 Citation

PhoneGo is those two papers in app form: the map-and-replay harness comes from PhoneCLI, and the open model behind the fallback path is OpenPhone. If either is useful to your research, please cite both.

@article{jiang2026phonecli,
  title={PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents},
  author={Jiang, Yangqin and Xu, Lingrui and Huang, Chao},
  journal={arXiv preprint arXiv:2609.35671},
  year={2026}
}

@inproceedings{jiang2026openphone, title={OpenPhone: Mobile Agentic Foundation Models}, author={Jiang, Yangqin and Huang, Chao}, booktitle={Findings of the Association for Computational Linguistics: ACL 2026}, pages={30362--30380}, year={2026} }


🔗 Related Projects

PhoneGo builds on PhoneCLI and OpenPhone, which in turn build upon excellent open-source projects. We sincerely thank their authors and contributors:

  • AndroidLab - The benchmark framework.
  • R1-V - Implementation details for the GRPO training methodology.
  • LLaMA Factory - The unified training framework enabling efficient model fine-tuning.
The CLI side of this repository (phonecli/, the evaluation suites, the training recipe) is unchanged; its original README is kept at docs/archive/README-phonecli-en.md.

📜 License

This project is released under the MIT License.

If this project helps you, please give us a Star🌟

🤖 Empower AI Phone with Agents!


❤️ Thanks for visiting ✨ PhoneGo!

Views

Chat with me