github-starred/android
2
0
Fork 0
mirror of https://github.com/bitwarden/android.git synced 2026-05-06 15:58:22 -05:00
Code Issues 1.1k Packages Projects Releases 122 Wiki Activity

[PR #6747] [PM-34487] llm: Add Android device interaction MCP server with ADB tooling #12983

New Issue
Open
opened 2026-04-11 03:55:21 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-04-11 03:55:21 -05:00
Owner
Copy Link

Original Pull Request: https://github.com/bitwarden/android/pull/6747

State: open
Merged: No

🎟️ Tracking

https://bitwarden.atlassian.net/browse/PM-34487

📔 Objective

Add a TypeScript MCP server and companion skill for structured Android device interaction via ADB. Replaces the initial shell script approach with proper XML parsing, structured dumpsys window parsing, and native geometry for obstruction detection.

🏗️ Architecture

Claude Code ──stdio──> MCP Server (TypeScript)
                         ├── tools/        (6 tool handlers)
                         ├── adb/          (execFile wrapper, device discovery)
                         ├── parsers/      (UIAutomator XML, dumpsys window)
                         └── geometry/     (bounds, obstruction, visible-region)

🔧 MCP Tools

Tool Description
capture Dump UI hierarchy XML and/or screenshot
find_element Find element by text/content-desc with two-layer obstruction detection
tap_element Find + tap + capture (auto-adjusts for obstructions)
tap_at Tap specific coordinates + capture
navigate Home, back, app-drawer navigation + capture
input_text Type text into focused field with shell-safe escaping

🛡️ Obstruction Detection

Two-layer system for detecting when UI elements are blocked:

  • Layer 1 — System overlays: Parses dumpsys window windows for TalkBack, PiP, accessibility overlays using touchable region bounds
  • Layer 2 — In-app elements: Finds topmost clickable element at tap point via XML tree traversal (catches FABs, dialogs, bottom sheets)

When obstructed, computes the largest visible strip (top/bottom/left/right of obstructor) and returns adjusted tap coordinates.

📁 Key Files

  • .mcp.json — MCP server config (stdio, auto-build)
  • .claude/mcp/android-device-server/ — TypeScript MCP server (self-contained for future plugin extraction)
  • .claude/skills/interacting-with-android-device/SKILL.md — Companion skill with tool docs and allowed-tools

✅ Testing

  • 57 unit tests across geometry, parsers, ADB discovery (fixture-based, no device required)
  • fast-xml-parser for UIAutomator XML → typed tree
  • Zod validation on all tool inputs
  • execFile (not exec) prevents host-side command injection
**Original Pull Request:** https://github.com/bitwarden/android/pull/6747 **State:** open **Merged:** No --- ## 🎟️ Tracking https://bitwarden.atlassian.net/browse/PM-34487 ## 📔 Objective Add a TypeScript MCP server and companion skill for structured Android device interaction via ADB. Replaces the initial shell script approach with proper XML parsing, structured `dumpsys window` parsing, and native geometry for obstruction detection. ## 🏗️ Architecture ``` Claude Code ──stdio──> MCP Server (TypeScript) ├── tools/ (6 tool handlers) ├── adb/ (execFile wrapper, device discovery) ├── parsers/ (UIAutomator XML, dumpsys window) └── geometry/ (bounds, obstruction, visible-region) ``` ## 🔧 MCP Tools | Tool | Description | |------|-------------| | `capture` | Dump UI hierarchy XML and/or screenshot | | `find_element` | Find element by text/content-desc with two-layer obstruction detection | | `tap_element` | Find + tap + capture (auto-adjusts for obstructions) | | `tap_at` | Tap specific coordinates + capture | | `navigate` | Home, back, app-drawer navigation + capture | | `input_text` | Type text into focused field with shell-safe escaping | ## 🛡️ Obstruction Detection Two-layer system for detecting when UI elements are blocked: - **Layer 1 — System overlays**: Parses `dumpsys window windows` for TalkBack, PiP, accessibility overlays using `touchable region` bounds - **Layer 2 — In-app elements**: Finds topmost clickable element at tap point via XML tree traversal (catches FABs, dialogs, bottom sheets) When obstructed, computes the largest visible strip (top/bottom/left/right of obstructor) and returns adjusted tap coordinates. ## 📁 Key Files - `.mcp.json` — MCP server config (stdio, auto-build) - `.claude/mcp/android-device-server/` — TypeScript MCP server (self-contained for future plugin extraction) - `.claude/skills/interacting-with-android-device/SKILL.md` — Companion skill with tool docs and `allowed-tools` ## ✅ Testing - 57 unit tests across geometry, parsers, ADB discovery (fixture-based, no device required) - `fast-xml-parser` for UIAutomator XML → typed tree - Zod validation on all tool inputs - `execFile` (not `exec`) prevents host-side command injection
GiteaMirror added the pull-request label 2026-04-11 03:55:21 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
app:authenticator
app:password-manager
bug
bug-passkey
customer-support
duplicate
enhancement
good first issue
help wanted
needs-reply
needs-response
pull-request
Mirrored from GitHub Pull Request
 question
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/android#12983
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?