shehbajdhillon/android-use

Overview

This skill controls Android devices over ADB to perform taps, swipes, typing, key presses, app launches, screenshots, and UI inspection. It uses the device's accessibility XML to read screen state and decide actions. It supports targeting specific devices with a serial flag for multi-device environments.

How this skill works

The skill runs ADB-backed shell scripts: check-device, get-screen (accessibility XML), tap, swipe, type-text, key, launch-app, install-apk, and screenshot. Workflow: verify device, dump UI XML, parse element attributes and bounds, compute tap coordinates, execute input scripts with -s <serial> when needed, then re-check the screen until the goal is reached. It uses element attributes like text, resource-id, content-desc, clickable, and bounds to choose actions safely.

When to use it

• Automate app navigation and testing on a connected Android device
• Perform UI interactions when remote control via ADB is acceptable
• Scripted installations or app launches during development
• Debug UI flows where accessibility XML can guide precise taps and typing
• Operate specific device when multiple devices/emulators are connected

Best practices

• Always run scripts/check-device.sh first and use -s <serial> if multiple devices are present
• Get a fresh UI dump with scripts/get-screen.sh before each decision to avoid stale state
• Calculate tap coordinates from bounds center and prefer elements with clickable="true"
• Wait 500ms–1s after actions for the UI to update and then re-read the screen
• Use screenshots when XML lacks visual context (images, colors, or non-accessible items)

Example use cases

• Open Chrome, focus the URL bar, type a query, and press Enter to search
• Launch Settings on a physical phone when an emulator is also connected by specifying the physical serial
• Scroll through a long list by checking visibility, swiping up until the target element appears, then tapping it
• Install an APK on a device and then launch the app to verify startup
• Dismiss permission dialogs by finding buttons like OK/Allow in the accessibility XML and tapping their centers