What was the modality of interpreting the UI/interacting with the phone via ADB? Did you/your agent use screenshots + vision model to essentially target coordinates on screen? I ask because I'm curious if you found a non-screenshot + coordinate based ADB method to do so (that doesn't rely on UIAutomator).
I wrote an APK that gives access to the accessibility service via an in-Android http server, which talks to a daemon running on the host machine which can, in turn, talk to Hermes. It's designed to be used with an emulator called Waydroid but you could probably run it on your device too. It does need to fall back to ADB when the accessibility tree is incomplete/inaccurate (which is often). And, going back to my question, I haven't found a means of interacting via ADB other than by coordinate/UIAutomator. To be clear, I am not smart or patient enough to be an Android dev, so I use it for ordering things on Amazon.
I didn't end up needing to go full scrcpy because my use case just runs Android in an emulator (Waydroid) for my Hermes agent to run apps. If I need vision or (x,y) coordinate interaction, I can just screenshot the emulator window and interact via ADB. However, I ended up using the accessibility tree to create a semantic, ephemeral ref API as the preferred mechanism for interaction. Basically more 'Playwright for Android' than 'Remote Desktop for Android'. But, unfortunately, the accessibility tree is often incomplete or inaccurate.
at work, an airline via an outsourcing company, the entire qa workflow is automated. we use claude desktop, the ai picks up a user story from azure devops via rest api or mcp, tests it on the app with playwright or claude browser using skills with software specifications for the product the user story will be merged into, then reports bugs and test results via mcp servers and rest apis to azure devops.
alas it needs manual intervention between each of those steps to copy and paste or publish results or comment on bugs because qa is outsourced and the outsourcing company refuses to not have someone do something manually in order to avoid losing revenue on their end.
the outsourcing company charges the client per person so they want to have as many people working as possible and the client, an airline, wants the exact opposite. they only hire software development students for the job so we are entirely able to fully automate the workflow and replace ourselves. people from the airline are too busy to do it themselves.
I am very sure most companies will just cobble together something with Claude and some skill.
I’d love to get some community feedback on a tool I’ve been building. It’s a fully AI-driven UX testing platform for mobile apps, designed to autonomously navigate and evaluate user experiences.
Check it out and let me know what you think. I’d really appreciate any feedback or suggestions. Thanks!
Just did this locally by giving my AI agent full ADB access to a test phone I have.
It's actually kinda amazing just watching it swipe, open the app launcher, and find issues to fix later.
What was the modality of interpreting the UI/interacting with the phone via ADB? Did you/your agent use screenshots + vision model to essentially target coordinates on screen? I ask because I'm curious if you found a non-screenshot + coordinate based ADB method to do so (that doesn't rely on UIAutomator).
I wrote an APK that gives access to the accessibility service via an in-Android http server, which talks to a daemon running on the host machine which can, in turn, talk to Hermes. It's designed to be used with an emulator called Waydroid but you could probably run it on your device too. It does need to fall back to ADB when the accessibility tree is incomplete/inaccurate (which is often). And, going back to my question, I haven't found a means of interacting via ADB other than by coordinate/UIAutomator. To be clear, I am not smart or patient enough to be an Android dev, so I use it for ordering things on Amazon.
By coordinate/ UIAutomator I'm assuming you mean the likes of scrcpy[0]?
Would be great if you have a way to do it programmatically.
https://github.com/Genymobile/scrcpy
I didn't end up needing to go full scrcpy because my use case just runs Android in an emulator (Waydroid) for my Hermes agent to run apps. If I need vision or (x,y) coordinate interaction, I can just screenshot the emulator window and interact via ADB. However, I ended up using the accessibility tree to create a semantic, ephemeral ref API as the preferred mechanism for interaction. Basically more 'Playwright for Android' than 'Remote Desktop for Android'. But, unfortunately, the accessibility tree is often incomplete or inaccurate.
https://www.djfa.ai/articles/clawdroid
at work, an airline via an outsourcing company, the entire qa workflow is automated. we use claude desktop, the ai picks up a user story from azure devops via rest api or mcp, tests it on the app with playwright or claude browser using skills with software specifications for the product the user story will be merged into, then reports bugs and test results via mcp servers and rest apis to azure devops.
alas it needs manual intervention between each of those steps to copy and paste or publish results or comment on bugs because qa is outsourced and the outsourcing company refuses to not have someone do something manually in order to avoid losing revenue on their end.
the outsourcing company charges the client per person so they want to have as many people working as possible and the client, an airline, wants the exact opposite. they only hire software development students for the job so we are entirely able to fully automate the workflow and replace ourselves. people from the airline are too busy to do it themselves.
I am very sure most companies will just cobble together something with Claude and some skill.
Some assembly required.
I’d love to get some community feedback on a tool I’ve been building. It’s a fully AI-driven UX testing platform for mobile apps, designed to autonomously navigate and evaluate user experiences.
Check it out and let me know what you think. I’d really appreciate any feedback or suggestions. Thanks!
app.deltix.ai
Okay this isn't half baked, it's basically raw. The app opens with everything greyed out. The support email bounces.
So much noise, might just put my head under rock for a few months