Let the AI operate the app
The app_* tools let the AI look at the window of a running application,
operate it and read its controls. The typical use is checking a UI change in the
app you are building: launch it, look at it, click through it, look again.
The tools are available on macOS only. On Windows and Linux they report that they are not supported.
Tools
| Tool | Effect | Raises the window | Mode |
|---|---|---|---|
app_windows | Lists the windows of running applications: id, process, title, position, size. With pid, only the windows of that process and of the processes it started. | no | any |
app_screenshot | Captures one window as an image. The window itself is captured, even if something lies on top of it. Scaled down to 1400 pixels on the longer side by default. | no | any |
app_elements | Lists the controls of a window from the accessibility interface: role, label, position, and a path handle. filter and pressable_only narrow the list. | no | any |
app_press | Presses a control by its path. The press goes to the control directly. | no | Brave |
app_menu | Chooses a menu command by path, for example ["File", "Save"]. A wrong path returns the entries available at that level. | no | Brave |
app_click | Clicks at a point in the window, left or right, single or double. | yes | Brave |
app_type | Types text into the control that has keyboard focus. Independent of the keyboard layout. | yes | Brave |
app_key | Presses a named key with modifiers, for example return, escape, cmd+s. | yes | Brave |
app_scroll, app_drag | Scrolls or drags inside the window. | yes | Brave |
Coordinates are points relative to the top left corner of the window. A point outside the window is refused.
Finding the window
npx tauri dev starts a chain of processes: npx, cargo, then the built program.
Only the last one owns a window. app_windows with the process id of the start
command follows this chain and returns the window.
A program that is still compiling has no window yet. An empty list means "not yet", not "failed".
Controls by name and by position
The AI prefers app_elements and app_press:
- A control keeps its name when the layout changes. A position does not.
- A list of controls costs a fraction of an image and can be searched.
app_pressandapp_menudo not bring the window to the front and do not take the keyboard focus. You can keep working in another window.
app_click, app_type, app_key, app_scroll and app_drag bring the target
window to the front first and check that it is in front. If it does not come to
the front, nothing is sent. While the AI works this way, typing in another
window can end up in the target window. Do not use the machine while the AI
clicks through an app.
Custom-drawn areas, games and many web views (including Electron apps such as VS Code) expose few or no controls. There the AI works with screenshots and clicks. A freshly started app may list few controls until its UI is built.
Permissions
macOS grants two permissions to Tracemoor, not to the AI:
| Permission | Needed for | Where |
|---|---|---|
| Screen Recording | Screenshots and window titles | System Settings › Privacy & Security › Screen & System Audio Recording |
| Accessibility | Clicks, keys, reading controls | System Settings › Privacy & Security › Accessibility |
Restart Tracemoor after granting a permission. Without Screen Recording,
app_windows returns the windows with empty titles and says so.
Working with it
- The AI should take a new screenshot or read the controls after every action. Whether a menu opened or nothing happened is only visible afterwards.
- A UI change counts as checked when the AI has seen it in the running app. Ask for that: "Launch the app and check the dialog in a screenshot."
- Run › Run / Launch starts the run configuration of the project. The AI can also start the app from the terminal and keep the process id.
Example request after a layout change:
Start the app with the dev run configuration, open the settings dialog and check in a screenshot that the labels are not cut off at the default window size.