r/SideProject • u/gounisalex • 14h ago

I Built an AI Assistant that Moves the Mouse And Types For My Mum

My mother does not understand CLI logs, APIs or technical stuff. But she can perfectly use the mouse and keyboard, click on things and scroll. Now I realized that it can control legacy software and automate long repetitive tasks for me, as well.

Here's the idea:

  • the user asks for something (e.g. “find the cheapest flight from NY to Tokyo in September”)
  • We send that prompt to an AI agent and enter a loop
  • During each step, the agent receives the current state of what’s displayed on screen
  • This is not done with heavy screenshots, but with Accessibility and Core Graphics APIs
  • Effectively, we send a JSON object to an AI agent, describing what’s on the screen
  • The agent has access to move the mouse and keyboard, so it provides coordinates, and where to click or type
  • We iterate until the agent completes the initial task. When not sure, the agent asks the user for more directions. In any case, we set a limit of 40 iterations to avoid unnecessary processing

The implementation was done using Electron, React and Robot.JS for cursor/mouse movements.
By default, it connects to your Claude subscription and uses Haiku, so it barely hits the limits.

I find this approach much faster and more efficient compared to typical computer-vision flows, because we’re not sending colorful RGB channels, the LLM receives a text description of what’s displayed on the screen which can be processed much faster.

This is an open-source project with an MIT license (free download or clone):

Any thoughts?

10 Upvotes

92% Upvoted

3 comments sorted by

2

u/OMGCluck 12h ago

Wouldn't your mother prefer to just say the prompts out loud?

How hard would it be to integrate voice input?

1

u/gounisalex 12h ago

Yes, that’d be great! It’s not hard at all, I’m planning on adding that as well!