Skip to main content
Version: Latest

Introduction to Model-Based GUI Automation

Prerequisites​

Before diving into these concepts, it helps to have:

  • Basic understanding of GUI automation (clicking, typing, finding elements)
  • Familiarity with the concept of "state" in software (helpful but not required)
  • No advanced mathematics required for this conceptual overview

For formal mathematical models, see Overall Model, States, and Transitions.

Overview​

Model-based GUI automation addresses the limitations of traditional methods by redefining the problem. Instead of creating sequential scripts (the process-based approach), the developer builds an explicit model of the GUI environment itself. This approach is inspired by principles from robotics, human cognition, and graph theory.

Process-Based vs. Model-Based: A Fundamental Shift​

To understand model-based GUI automation, we must first understand what it replaces.

Process-Based Automation (Traditional Approach)​

In process-based GUI automation, you write sequential scripts that specify exact steps:

1. Click login button
2. Wait 2 seconds
3. Type username
4. Type password
5. Click submit
6. Wait for dashboard
7. Click settings
8. ...

The Problem: If step 3 fails (button moved, network delay, unexpected popup), the entire script breaks. The automation has no understanding of where it is or how to recover. It only knows the sequence of steps.

This is like giving someone directions as: "Walk 100 steps forward, turn left, walk 50 steps..." If they encounter a closed door at step 75, they're stuckβ€”they have no map to find an alternative route.

Model-Based Automation (Brobot's Approach)​

In model-based automation, you create a map of the GUI environment (states, transitions, elements), and the framework finds its own path to the goal.

// Define the map (domain knowledge)
@State(name = "LoginPage")
public class LoginState {
    @StateImage private Image loginButton;
}

@State(name = "Dashboard")
public class DashboardState {
    @StateImage private Image logo;
}

// Request navigation - framework finds the path
stateNavigator.openState("Dashboard");

The Advantage: If the direct path fails, the framework can find an alternative route because it understands the GUI's structure. It knows where it is (active states) and what paths exist to the destination.

This is like giving someone a map. If they encounter a closed door, they can look at the map and find an alternative route.

The Explicit Model: A Digital Twin​

At its core, model-based GUI automation transforms the developer's implicit mental model of the GUI into an explicit, machine-readable format. This explicit model, called the State Structure (Ξ©), is a structured representation that contains:

  • States (S): Distinct configurations of the GUI (like "LoginPage", "Dashboard", "SettingsMenu")
  • Elements (E): Visual features that identify each state (images, regions, text patterns)
  • Transitions (T): Actions that move between states (click login button β†’ Dashboard)

Think of the State Structure as a roadmap where:

  • States are destinations (cities)
  • Elements are landmarks that identify each destination
  • Transitions are the roads connecting destinations

This applied model functions as a digital twin of the GUI environment. It is a virtual representation that can predict and interact with the digital environment in the same way a traditional digital twin represents a physical system.

Example: When you mentally navigate a familiar application, you think "I'm on the home screen, I need to click settings to reach the settings menu." The State Structure externalizes this mental model into code that Brobot can use to navigate autonomously.

Domain vs. Strategic Knowledge​

The framework is built on a separation of knowledge into two distinct categories, a concept adapted from studies of human problem-solving:

1. Domain Knowledge (Ξ©) - The "Map"​

What it is: Specific information about your GUI environment being automated.

Contains:

  • States: What screens/configurations exist ("LoginPage", "Dashboard", "Settings")
  • Elements: What visual features identify each state (images, buttons, text)
  • Transitions: What actions connect states (clicking "Settings" button moves from Dashboard to Settings)

Your responsibility: Define these for your specific application using Brobot annotations.

Example in Brobot:

@State(name = "LoginPage")
public class LoginState {
    @StateImage
    private Image loginButton = new StateImage.Builder()
        .addPattern("login-button.png")
        .build();

    @StateImage
    private Image usernameField = new StateImage.Builder()
        .addPattern("username-field.png")
        .build();
}

@TransitionSet(state = LoginState.class)
public class LoginTransitions {
    @OutgoingTransition(activate = {DashboardState.class})
    public boolean login() {
        action.type(usernameField, "myuser");
        action.type(passwordField, "mypass");
        return action.click(loginButton).isSuccess();
    }
}

2. Strategic Knowledge (F) - The "Navigator"​

What it is: Problem-agnostic algorithms for understanding and manipulating any GUI environment.

Includes:

  • Pathfinding: Finding routes through the State Structure graph to reach a target state
  • Path Traversal: Executing transitions along a chosen path, handling failures
  • State Management: Tracking which states are currently active based on what's visible
  • Action Execution: Performing low-level operations (click, type, find) with error handling

Framework's responsibility: Brobot implements these algorithmsβ€”you don't write pathfinding code.

Example Usage:

// You simply request the destination
stateNavigator.openState("SettingsPage");

// Framework handles:
// 1. "Where am I?" (State Management)
// 2. "What paths exist to Settings?" (Pathfinding)
// 3. "Try shortest path, if fails try alternative" (Path Traversal)
// 4. "Execute each transition's actions" (Action Execution)

Why This Separation Matters​

For developers: You focus on defining your GUI's structure (states, images, transitions) and your business logic. The framework handles the complex navigation algorithms.

For robustness: When the GUI changes, you update only the affected state or transition definition. The pathfinding logic remains unchanged and automatically adapts to the new structure.

Analogy:

  • Domain Knowledge = Updating a city's map when new roads are built
  • Strategic Knowledge = GPS navigation algorithm that works with any map

This separation allows the automation developer to focus on defining their application's GUI (the domain) and business logic, while the framework handles the complex strategies for navigation and interaction.

Handling Uncertainty: Stochasticity in GUI Environments​

Real GUI environments are non-deterministic. Actions don't always succeed:

  • Network requests cause unpredictable delays
  • Animations have variable timing
  • Elements may appear in slightly different positions
  • Unexpected popups can appear at any time
  • Pattern matching has inherent probability (similarity thresholds)

In formal terms, environmental stochasticity (Θ) represents all external events and factors that can unpredictably influence the GUI state during automation.

Why This Matters​

In process-based automation, uncertainty breaks scripts:

Step 1: Click button β†’ SUCCESS
Step 2: Wait 2 seconds β†’ (popup appears!)
Step 3: Type in field β†’ FAILS (popup covers field)
Step 4: All subsequent steps FAIL

In model-based automation, the State Management system detects unexpected states:

// Attempt transition to Dashboard
stateNavigator.openState("Dashboard");

// If unexpected popup appears:
// 1. State Management detects popup is active
// 2. Path Traversal checks for transitions from {Dashboard, Popup}
// 3. Framework executes "close popup" transition
// 4. Resumes navigation to Dashboard

Key Insight: By maintaining explicit knowledge of active states and possible transitions, the framework can detect when the GUI diverges from expectations and dynamically adapt its path.

Probabilistic Pattern Matching​

When Brobot searches for an image on screen, success is probabilistic:

  • 95% similarity threshold β†’ image found with 97% match β†’ SUCCESS
  • 95% similarity threshold β†’ image found with 92% match β†’ FAILURE
  • Image occluded by popup β†’ not found β†’ FAILURE

The State Structure includes multiple ways to identify each state, providing redundancy when one method fails due to stochasticity.

The Visual API​

The combination of the user-defined state structure (Ξ©) and the framework (F) creates what the paper calls a Visual API. This API abstracts away the complexities of visual automation, allowing the user's application to interact with the GUI at a high level, similar to how a programmatic API works.

High-Level vs. Low-Level Operations​

Traditional (Low-Level):

// You must specify exact pixel coordinates and handle errors
robot.click(450, 320);
Thread.sleep(2000);
if (!findImage("dashboard.png", 0.85)) {
    // Manually handle error - what now?
    throw new Exception("Dashboard not found");
}
robot.click(800, 150);  // Settings button
// If popup appears, entire script breaks

Visual API (High-Level):

// Simply request the destination state
stateNavigator.openState("SettingsPage");

// Visual API handles:
// - Finding current location (active states)
// - Pathfinding to Settings
// - Image recognition for all elements
// - Clicking correct locations
// - Handling unexpected popups/delays
// - Retrying failed transitions
// - Finding alternative routes if needed

Why "Visual API"?​

Just as REST APIs abstract server operations:

// High-level API call
api.getUserProfile(userId);
// Abstracts: HTTP requests, JSON parsing, error handling, retries

The Visual API abstracts visual automation:

// High-level state navigation
stateNavigator.openState("Dashboard");
// Abstracts: image recognition, pathfinding, state tracking, error recovery

The key difference: The Visual API works with any GUI through visual pattern matching, not just applications with programmatic APIs.

Why Developers Should Care​

Model-based GUI automation solves real pain points:

1. Reduced Code Complexity​

  • Process-based: 500 lines of sequential click/wait/type commands
  • Model-based: 50 lines defining states + 1 line: navigateTo("Goal")

2. Robustness to GUI Changes​

  • Process-based: GUI changes break entire script, requires full rewrite
  • Model-based: Update single state definition, pathfinding adapts automatically

3. Error Recovery​

  • Process-based: Script stops on first failure
  • Model-based: Framework finds alternative paths, recovers from unexpected states

4. Scalability​

  • Process-based: Complexity grows exponentially with features
  • Model-based: Complexity grows linearly (define states, framework handles paths)

5. Testability​

  • Process-based: Difficult to unit test sequential scripts
  • Model-based: States and transitions are independently testable

Example Impact: A 30-state GUI has potentially trillions of paths in process-based automation (exponential). In model-based, you define ~150 transitions (30 states Γ— 5 transitions each = polynomial), and the framework handles all path combinations.

For deeper understanding, explore:

Notation Guide​

Throughout the theoretical foundations documentation, you'll see mathematical notation from the academic paper:

  • Ξ© (Omega) = State Structure (the map of your GUI)
  • F = Framework (Brobot's implementation of strategic knowledge)
  • Θ (Theta) = Stochasticity (environmental uncertainty)
  • Ξ (Xi) = Visible GUI (what's currently on screen)
  • S = Set of all states
  • E = Set of all elements
  • T = Set of all transitions
  • M = State Management system
  • a = Action Model
  • Ο„ (tau) = Transition Model
  • Β§ = Path Traversal Model

These symbols provide precise mathematical definitions in the formal model documents. For conceptual understanding, focus on the ideas rather than the notation.