Graphic User Interface (GUI) automation

MouseRobot "sees" graphic objects on the screen. With MouseRobot you can press a button, select a list item, set text to a field, move windows, automate filling form (including WEB-forms).

We recommend to follow the recommendations described in the article for successful automation of GUI programs.

1. Be careful when choosing the responsive to mouse objects

Some control elements can react to mouse-over, changing its form, name, size or other properties. This can interfere with the normal performance of the recorded task, as MouseRobot will remember and seek not the original item but its altered form, which appears only when the mouse points to it.


One second after the mouse cursor is over the list item of Services hint appears. It is not displayed near the mouse cursor, but directly beneath it.

Sample of hint appearence

As a result, if you press Ctrl to remember the properties of an element after the appearance of the hint, MouseRobot remembers it. In the process of task execution the hint will not on the screen and MouseRobot will not able to execute the step.

2. Use "hot keys"

If it is possible to call the necessary command, function, dialogue or operation in the automated program by pressing key combinations, it is recommended to use them.

sample of using hotkey

3. Move between the data entry fields using the keyboard

To move between the dialog fields, table cells, etc. use the Tab key and Shift + Tab, to confirm or cancel data entry press {ENTER} and {ESC} (or other, depending on the automated program). In this case, it is recommended to specify heading of dialog or window as a subject in the "Search object on screen" step and move to its elements by pressing keys using "Mouse and keybord" step. This will allow obtaining compact and easily modifiable task records in MouseRobot, and moreover, increasing the reliability of execution.

Dialog sample Filling dialog fields sample

4. Use quick transition to elements of the window with Alt + letter

Many programs allow the transfer to the necessary control element (flags, buttons, menu items, ...) by pressing the keyboard shortcut Alt + letter (when pressing Alt the corresponding letters in the names of the elements are displayed as underlined).
Quick transition sample

5. Automate the search for headings of fields

If the window of the automated program contains a large number of similar fields (e.g., web-form) it is desirable to search not for the fields, but for their headings, which in most cases are unique and can be found with a high degree of reliability. If necessary, the input focus can be switched from the header to the field by pressing the {TAB}, however, most programs will automatically activate the field when you click on its title.

Window with similar fields sample

6. Choose easily recognizable objects

When there are a large number of similar objects, you can improve the reliability of the search facility by entering an object of a higher level. You can navigate through the levels by pressing the PgUp and PgDn in the capture of the object (for example, to specify the entire table instead of a specific cell).


The toolbar of the graphics editor Paint.NET contains the drawing tools but does not provide the names of the keys or their ordinal numbers. As a result, all the buttons look identical to MouseRobot, which can result in incorrect task execution. However, the buttons of the toolbar will never change their position, so when selecting an object it is sufficient to press PgUp to rise from the button to the toolbar, and remember the position of the mouse press already relating to it.

Paint.Net toolbar

7. Use browsers that support advanced automation

It is recommended to use Microsoft Internet Explorer and Mozilla Firefox browser when automating the web-interface, as they provide access to page elements for external programs, such as MouseRobot. Similar devices for Google Chrome, Apple Safari and Opera are in development and should be available in the latest versions of these browsers.

8. Extend timeout for step "Search object" following after the program launch

In reality, the time required to load the program and the contents of its windows may differ from the average by several times.

If the execution of the previous step may take some time (running an application, loading a web page), then set "Step Timeout" parameter for the following steps (the "Common" panel) for 10 seconds or more.

9. Check for running copy of the launched program instead of restarting it

Often it is preferable to use the already running program instead of starting a new copy. You can use the "Check and run if needed" snippet.

10. Use looping facilities

MouseRobot provides an excellent opportunity to automate the repetitive operations of the same elements. You can use snippets from "Loop through objects"

11. Monitor the performance of long processes

If the automated program displays a dialog with progress of execution (for example convert video, compilation, etc.), while an action is being performed, it is recommended to add the snippet "Waiting for window close" to the task.

Waiting for Window Close snippet

© AutomationBox 2005-2011