Marcio Miike Jr. / Notes

An agent that works a screen

Most software has no door for an AI to come in by. It has a screen. An agent can use the screen the way a person does: look, decide on one action, do it, and look again.

once the screen has changed, so look again when finished success or failure trains the model Instruction in plain words Look takes a screenshot Decide picks one action Act click, type, scroll Screen any application Done nothing left to do 1 2 3 once the screen has changed, so look again success or failure trains the model Instruction in plain words Look takes a screenshot Decide picks one action Act click, type, scroll Screen any application Done nothing left to do 1 2 3
The agent never plans the whole task and runs it blind. It takes one action, then looks at the screen again, because the screen is the only proof of what happened.

Reading the picture

The instruction is given once, in plain words. From then on the agent goes around the loop. It takes a screenshot. A model that reads images and text looks at it and picks one action: click here, type this, scroll down. The action is carried out, and the screen changes.

The green line is the point. After every action the agent looks again. A button may have moved, a dialog may have opened, the click may have missed.

When the model sees nothing left to do, it stops. Whether the task succeeded is then used to train the model, so the next run decides better.

Where it goes wrong

  • 1The lookThe model misreads the screen. Small icons and dense tables are the usual cause.
  • 2The decisionThe model picks an action that looks reasonable and is wrong, such as the Save button of the wrong form.
  • 3The loopSmall errors add up. An agent that is right 95 times in 100 on each step finishes a 20-step task only about 36 times in 100.

What I built

From March to May 2025 I led the building of an early version, as project manager and engineer. It used a vision-language model for the look and the decision, trained further by reinforcement learning. For the demo, the actions were carried out in a browser through Playwright.

It worked on ordinary office tasks, and it was early. The third problem above is the one that decides whether such an agent is useful.