Transform any Android device into a Home Assistant-integrated automation platform.
Visual Mapper is an open-source system that lets you monitor, control, and automate Android devices directly from Home Assistant. Create sensors from any app's UI, automate device interactions, and integrate legacy Android-only devices into your smart home.
Use Cases:
- Create Home Assistant sensors from any Android app's UI (battery, media status, notifications)
- Automate repetitive tasks on tablets, phones, or Android-based devices
- Build custom dashboards that interact with Android devices
- Control legacy devices that only have Android apps (thermostats, cameras, etc.)
Visual Mapper uses a hybrid architecture with three components:
+-------------------+ HTTP/MQTT +-------------------+
| Android App | <----------------> | Python Server |
| (Companion App) | | (FastAPI) |
+-------------------+ +-------------------+
| |
| Accessibility | MQTT
| Service |
v v
+-------------------+ +-------------------+
| Android Device | | Home Assistant |
| UI Elements | | |
+-------------------+ +-------------------+
| Component | Role | Technology |
|---|---|---|
| Python Server | Orchestrates flows, manages sensors, serves web UI | FastAPI, ADB, MQTT |
| Android Companion App | Captures UI state, executes actions locally | Kotlin, Accessibility Service |
| Home Assistant Add-on | Easy deployment and integration | Docker |
- Python 3.11+
- Android device with Developer Options enabled
- Both devices on the same network
git clone https://github.com/botts7/visual-mapper.git
cd visual-mapper/backend
pip install -r requirements.txtpython main.pyWeb UI available at http://localhost:8080
Option A: WiFi ADB (Recommended)
- Enable Developer Options on your Android device
- Enable Wireless Debugging (Android 11+)
- Navigate to
http://localhost:8080/devices.html - Use Network Discovery or enter device IP manually
Option B: USB ADB
- Enable USB Debugging on your Android device
- Connect via USB cable
- Device will appear automatically in the web UI
| Feature | Description |
|---|---|
| Screenshot Capture | Real-time device screenshots with element detection |
| Device Control | Tap, swipe, type, scroll on devices |
| Sensor Creation | Create HA sensors from any UI element |
| MQTT Integration | Auto-discovery and state publishing to Home Assistant |
| Flow Automation | Record and replay multi-step interactions |
| Flow Wizard | Visual step-by-step flow creation |
| Smart Flows | AI-assisted flow generation from app screens |
| Multi-Device | Manage multiple Android devices |
| WiFi ADB | Wireless connection (Android 11+) |
| Network Discovery | Auto-scan for Android devices |
| Live Streaming | Real-time device screen streaming |
| Dark Mode | Theme toggle with system preference |
visual-mapper/
├── backend/
│ ├── main.py # FastAPI server entry point
│ ├── core/ # Core modules (ADB, MQTT, sensors, flows)
│ ├── routes/ # API route handlers
│ ├── services/ # Business logic
│ ├── utils/ # Utility modules
│ └── Dockerfile # Container build
├── frontend/
│ └── www/ # Web UI (HTML, CSS, JS)
├── config/ # Configuration files
└── README.md
| Repository | Description |
|---|---|
| visual-mapper | Main server application (this repo) |
| visual-mapper-android | Android companion app |
| visual-mapper-addon | Home Assistant add-on |
Copy .env.example to .env and customize:
cp .env.example .envKey settings:
MQTT_BROKER=localhost
MQTT_PORT=1883
CONNECTION_CHECK_INTERVAL=30
CONNECTION_RETRY_ENABLED=truecd backend
docker build -t visual-mapper .
docker run -p 8080:8080 --network host visual-mapperContributions welcome! Please:
- Fork the repository
- Create a feature branch
- Submit a pull request
MIT License - see LICENSE for details.
- Issues: GitHub Issues
Version: 0.2.74 | License: MIT