Testing the Automation

Building on the Overall Model

This document focuses on testing the automation code itself, one of the most significant innovations enabled by model-based GUI automation. Before diving into testing concepts, ensure you understand:

• Overall Model - The complete formal model (Ξ, Ω, a, M, τ, §)
• States - State Structure and state management
• Transitions - Transition model and path traversal

Understanding how to test your automation is critical for building reliable, maintainable GUI automation applications.

The Fundamental Innovation: Testing Automation

Testing Automation vs. Automated Testing

One of the principal innovations of model-based GUI automation is the ability to systematically test the automation code itself. This introduces a critical distinction:

Automated Testing (Traditional):

• Using software to test another application (the Application Under Test, or AUT)
• Metaphor: Building a robot to test a car's brakes
• Focus: The car's performance
• Example: Using Selenium to test a web application

Testing Automation (Novel):

• Testing the automation code itself to ensure it functions correctly
• Metaphor: Testing the robot to ensure its own arms, sensors, and logic work before it touches the car
• Focus: The robot's internal mechanisms
• Example: Testing your Brobot automation code to verify it navigates states correctly

From the Paper: "Testing automation, a novel capability enabled by the model-based approach, refers to the process of testing the automation code itself. In the metaphor, this is akin to testing the robot to ensure its own arms, sensors, and logic function correctly before it ever touches the car."

Why This is Revolutionary

Model-based GUI automation makes testing automation feasible for the first time in a structured way, applying standard software engineering practices to a domain where they were previously considered impractical or impossible.

Historical Challenge: Traditional GUI automation couldn't be tested because:

• Live GUI environments are stochastic (unpredictable)
• Setting up test environments requires the actual application running
• Visual elements change in ways that break reproducibility
• No way to run tests in CI/CD pipelines without full GUI setup

Model-Based Solution: Separate the automation logic from the live GUI through:

• Mock execution using recorded interaction histories
• Fixed scenes (screenshots) for deterministic pattern matching
• Explicit state models that can be validated independently

This elevates GUI automation development to the same engineering rigor as other software domains, enabling:

• ✅ Unit testing of individual components
• ✅ Integration testing of complete workflows
• ✅ CI/CD pipeline integration
• ✅ Test-driven development (TDD)
• ✅ Continuous validation of the state model

From the Paper: "Model-based GUI automation makes this second type of testing feasible for the first time in a structured way, applying standard software engineering practices to a domain where they were previously impractical."

The Testing Workflow

Before diving into formal definitions, understand the practical workflow:

Phase 1: Record Action History

During normal automation runs (with live GUI):

1. Automation executes actions against live GUI (Ξ)
2. Each action records its parameters, active states, and results
3. Results are saved as Action Snapshots (AS)
4. Collection of snapshots forms Action History (AH)
5. Action History is saved to JSON files

Example Recording:

Action: CLICK on "Login Button"
Active States: {LoginScreen}
Result: SUCCESS
Match Location: (450, 320)
Duration: 0.15s
→ Saved to: login_button_AS.json

Phase 2: Test with Mock Execution

During test runs (no live GUI needed):

1. Load Action History from JSON files
2. Automation executes in mock mode
3. When action is requested, framework finds matching historical snapshot
4. Historical results are returned as if action was performed
5. State Management and Path Traversal work identically to live mode

Example Mock Execution:

Test requests: CLICK on "Login Button" with states {LoginScreen}
Framework searches AH for matching snapshot
Finds: login_button_AS.json with matching parameters
Returns: SUCCESS result from history
State Management updates: {LoginScreen} → {Dashboard}
Test continues without touching live GUI

This workflow enables reproducible testing without environmental stochasticity.

Integration Testing Through Mocking

Integration testing validates that the entire automation system—states, transitions, path traversal, state management—works together correctly. In model-based GUI automation, this is achieved through mocking.

The Concept

During mock runs, the automation does not interact with the live GUI environment (Ξ, Θ). Instead, it operates on a predefined Action History (AH) containing recorded outcomes of previous GUI interactions.

Key Insight: By replacing the live GUI with historical data, we can test the entire system's behavior deterministically, without stochastic interference from the actual GUI.

Formal Definitions

Atomic Action (unchanged):

a = (o_a, E_a, ζ)

Mock Action Function:

f_a^mock: (a, S_Ξ) → (r_a, S_a)

where:

• a: The action to be performed
• S_Ξ: Current set of active states (replaces Ξ and Θ from live function)
• r_a: Action result (returned from history)
• S_a: State information (changes to active states)

Comparison to Live Action Function:

Live:  f_a: (a, Ξ, Θ) → (Ξ', r_a)
Mock:  f_a^mock: (a, S_Ξ) → (r_a, S_a)

Critical Difference: Mock function takes active states (S_Ξ) as input instead of the live GUI (Ξ) and environmental stochasticity (Θ). It returns action results without modifying any actual GUI.

Action History Structure

GUI Element with Action History:

e ∈ E_a = (o_e, AH)

where:

• o_e: Element parameters (e.g., filename for images, bounding box for regions)
• AH: Action History for this element

Action History Set:

AH = {AS₁, AS₂, ..., ASₙ}

A set of Action Snapshots recorded during previous live runs.

Action Snapshot (AS):

AS = (o_a^h, S_Ξ^h, r_a^h)

where (^h denotes "historical"):

• o_a^h: Historical action parameters (e.g., action type, element parameters)
• S_Ξ^h: Active states when the action was recorded
• r_a^h: Recorded action results (success/failure, match locations, duration)

Structural Equivalence:

For a historical snapshot to match a current action request:

o_a^h ≅ o_a    (action parameters are compatible)
S_Ξ^h ≅ S_Ξ    (active states match)
r_a^h ≅ r_a    (results are structurally compatible)

The ≅ symbol denotes structural equivalence—the components have compatible types and semantics even if not identical.

How Mock Execution Works

The Two Key Functions:

1. Matching Function: Identifies relevant historical actions

   match: (a, S_Ξ, AH) → AS ∪ {∅}

   Compares:

   • Current action parameters (o_a) to snapshot parameters (o_a^h)
   • Current active states (S_Ξ) to snapshot active states (S_Ξ^h)

   Returns matching snapshot or ∅ if no match found.

2. Selection Function: Chooses a snapshot from matching candidates

   select: P(AS) → AS

   When multiple matches exist, selection can be:

   • Deterministic: First match, best match by similarity score
   • Stochastic: Random selection weighted by historical frequency

Mock Action Execution:

r_a := r_a^h and S_a := f(r_a^h)

The framework retrieves historical results and uses them as the current action's results.

Computational Isomorphism

From the Paper: "The structural equivalence between mock outputs and live automation ensures that both the action snapshot (AS) and the live action function (f_a) produce results (r_a) and state information (S_a) in identical formats, preserving computational isomorphism across testing and production environments."

What This Means:

Downstream components—State Management (M), Path Traversal (§), Transition Function (f_τ)—cannot distinguish between:

• Results from live GUI interactions
• Results from mock historical data

This isomorphism (same structure) allows the entire automation system to run identically in both modes, enabling true integration testing.

Practical Benefit: Tests verify the same code paths, logic, and state transitions that execute in production.

Visual Representation

┌─────────────────────────────────────────────────────────┐
│                  Live Automation                         │
├─────────────────────────────────────────────────────────┤
│ Action Request (a)                                       │
│         ↓                                                │
│ f_a(a, Ξ, Θ) → (Ξ', r_a)                               │
│         ↓                                                │
│ State Management updates S_Ξ                            │
└─────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│                  Mock Automation                         │
├─────────────────────────────────────────────────────────┤
│ Action Request (a) + Active States (S_Ξ)               │
│         ↓                                                │
│ match(a, S_Ξ, AH) → AS = (o_a^h, S_Ξ^h, r_a^h)        │
│         ↓                                                │
│ select(matching AS) → chosen AS                         │
│         ↓                                                │
│ f_a^mock(a, S_Ξ) → (r_a^h, S_a)                        │
│         ↓                                                │
│ State Management updates S_Ξ (same as live!)            │
└─────────────────────────────────────────────────────────┘

Both paths result in identical state transitions, enabling true integration testing.

Unit Testing

While integration testing verifies system-level behavior, unit testing requires deterministic verification of individual components. The challenge: some actions (pattern matching) must be validated, while others (clicks, typing) can be simulated.

The Hybrid Approach

The unit testing model uses two different action functions to ensure reproducible results:

For Observation Actions (FIND, MATCH):

f_a^unit: (a, Ξ_x) → (r_a, S_a)

where:

• Ξ_x: A fixed scene (e.g., a static screenshot)
• Observation actions perform real pattern matching on static images
• Results are deterministic because the scene never changes

For All Other Actions (CLICK, TYPE, MOVE):

f_a^mock: (a, S_Ξ) → (r_a, S_a)

where:

• Results are simulated using historical data from Action Snapshots
• Interactive actions don't need validation—they're deterministic given their inputs

From the Paper: "Observation actions perform pattern matching on static screenshots, while interactive actions such as clicks and keyboard inputs are simulated using predetermined results from action snapshots. This hybrid approach enables fully reproducible tests that validate real image recognition functionality while deterministically controlling actions that inherently require a live GUI environment."

Why This Hybrid Approach?

Observation Actions (image recognition):

• Core functionality that MUST be validated
• Pattern matching algorithms need testing
• Different similarity thresholds produce different results
• Solution: Run against fixed screenshots for reproducibility

Interactive Actions (clicks, typing):

• Behavior is deterministic given inputs
• Don't need live GUI to validate
• Executing them requires actual GUI environment
• Solution: Mock their results using historical data

Comparison: Integration vs. Unit Testing

Aspect	Integration Tests	Unit Tests
Results Source	Historical action data from previous GUI interactions	Static screenshots for observations, selected historical actions for others
Stochasticity	Preserved through sampling of historical results	Eliminated using fixed inputs for reproducibility
Purpose	Verify system behavior and state transitions	Test individual components in isolation
Scope	Complete workflows (e.g., "login → navigate → save data")	Individual actions or states (e.g., "find login button")
Test Speed	Fast (no GUI required)	Very fast (deterministic, no randomness)
Use Case	CI/CD pipeline smoke tests, regression testing	TDD, component validation, debugging

Visual Representation: Hybrid Testing

┌─────────────────────────────────────────────────────────┐
│              Unit Test: Observation Action               │
├─────────────────────────────────────────────────────────┤
│ Action: FIND(loginButton)                                │
│         ↓                                                │
│ f_a^unit(a, Ξ_x) where Ξ_x = screenshot.png            │
│         ↓                                                │
│ REAL pattern matching on static image                   │
│         ↓                                                │
│ Result: Match at (450, 320) with 97.5% similarity       │
│ ✓ Validates pattern matching works correctly            │
└─────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│            Unit Test: Interactive Action                 │
├─────────────────────────────────────────────────────────┤
│ Action: CLICK(450, 320)                                  │
│         ↓                                                │
│ f_a^mock(a, S_Ξ) → retrieve from AH                     │
│         ↓                                                │
│ Result: SUCCESS (from historical data)                  │
│ ✓ Click behavior simulated deterministically            │
└─────────────────────────────────────────────────────────┘

Model Validation Through Testing

Testing in model-based GUI automation serves a dual purpose:

1. Validate automation instructions (the business logic)
2. Verify the State Structure (Ω) itself

This second purpose is unique and powerful. Your tests don't just check if your code works—they validate whether your model of the GUI is accurate.

What Gets Validated

Mock runs provide insights into each component of Ω = (E, S, T):

Elements (E):

• Tests verify that defined GUI elements are sufficient for navigation
• Missing elements are revealed when actions fail to find required patterns
• Example: Test fails → "Login button not found" → Add button image to LoginState

States (S):

• Mock runs validate state definitions
• Reveal missing states or redundant state definitions
• Example: Test expects Dashboard after login, but state not defined → Add DashboardState

Transitions (T):

• Testing exposes gaps or errors in transition paths
• Identifies missing transitions between states
• Example: No path from Settings to Profile → Add transition

Failure Analysis

From the Paper: "For example, consider a mock run failure when attempting to reach state s_t ∈ S from state s₀ ∈ S. This could indicate: 1. Missing transition: ∄ t ∈ T such that t connects s₀ to s_t; 2. Incomplete element set: ∃ e ∉ E required for transition; 3. State definition error: s_t or intermediate states are inadequately defined."

When a test fails, the error points to specific model issues:

1. Missing Transition:

Formal: ∄ t ∈ T such that t connects s₀ to s_t
Plain: No transition exists from current state to target state
Fix: Add transition defining the action sequence to connect states

Example:

Test: Navigate from LoginScreen to ProfileSettings
Error: "No path found from LoginScreen to ProfileSettings"
Analysis: Missing transitions - must go through Dashboard first
Fix: Add transitions:
  - LoginScreen → Dashboard (via login)
  - Dashboard → ProfileSettings (via settings menu)

2. Incomplete Element Set:

Formal: ∃ e ∉ E required for transition
Plain: An element needed for the transition is not defined in the model
Fix: Add the missing element to the appropriate state

Example:

Test: Execute transition Dashboard → Settings
Error: "Element 'settingsButton' not found in Dashboard state"
Analysis: Settings button image not included in Dashboard state definition
Fix: Add settingsButton image to Dashboard state

3. State Definition Error:

Formal: s_t or intermediate states are inadequately defined
Plain: The target state or states along the path are not properly defined
Fix: Improve state definition with more/better elements

Example:

Test: Verify Dashboard state is active
Error: "Dashboard state not detected" (state images not found on screen)
Analysis: Dashboard state only has logo image, but logo is occluded by popup
Fix: Add multiple identifying elements (logo, menu bar, widgets) to Dashboard

Iterative Model Improvement

From the Paper: "This iterative improvement through testing enables the development of increasingly accurate models of the GUI environment, supporting more sophisticated automation tasks."

The testing workflow naturally leads to model refinement:

1. Write test for desired behavior
2. Run test → FAIL (model incomplete)
3. Analyze failure → Identify missing component
4. Add element/state/transition to model
5. Run test → PASS
6. Model is now more complete
7. Repeat for more complex behaviors

This is test-driven development for GUI automation, where tests guide the construction of an accurate GUI model.

Practical Implementation in Brobot

Setting Up Tests with BrobotTestBase

All Brobot tests should extend BrobotTestBase, which automatically configures mock mode:

import io.github.jspinak.brobot.test.BrobotTestBase;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

public class LoginAutomationTest extends BrobotTestBase {

    @Test
    public void testLoginFlowWithMockHistory() {
        // BrobotTestBase automatically:
        // - Enables mock mode
        // - Sets fast mock timings
        // - Configures headless environment support

        // Your test code here
        boolean loginSuccess = stateNavigator.openState("Dashboard");

        assertTrue(loginSuccess);
        assertTrue(stateMemory.isActive("Dashboard"));
    }
}

What BrobotTestBase Provides:

• Automatic mock mode activation via BrobotProperties
• Headless environment compatibility (works in CI/CD)
• Fast mock timings (0.01-0.04 seconds for operations)
• Consistent test configuration

Recording Action History

Method 1: Automatic Recording During Live Runs

Configure in application.properties:

# Enable action history recording
brobot.action.history.record=true
brobot.action.history.path=src/test/resources/histories/

# Record all actions for specified states
brobot.action.history.states=Login,Dashboard,Settings

During live automation, all actions are automatically recorded to JSON files.

Method 2: Programmatic Recording with StateImage.Builder

import io.github.jspinak.brobot.model.state.StateImage;
import io.github.jspinak.brobot.mock.ActionRecord;

StateImage loginButton = new StateImage.Builder()
    .addPatterns("login-button.png")
    .withActionHistory(
        new ActionRecord.Builder()
            .setActionType(ActionType.CLICK)
            .setSuccessRate(0.95)
            .setAverageMatchLocation(450, 320)
            .setAverageDuration(0.15)
            .build()
    )
    .build();

Method 3: Using MockActionHistoryBuilder

import io.github.jspinak.brobot.mock.MockActionHistoryBuilder;

StateImage reliableButton = new StateImage.Builder()
    .addPatterns("button.png")
    .withActionHistory(
        MockActionHistoryBuilder.builder()
            .successRate(1.0)  // Always succeeds
            .matchRegion(new Region(400, 300, 100, 50))
            .averageDuration(0.1)
            .build()
    )
    .build();

StateImage flakyElement = new StateImage.Builder()
    .addPatterns("flaky-element.png")
    .withActionHistory(
        MockActionHistoryBuilder.builder()
            .successRate(0.7)  // Succeeds 70% of the time
            .matchRegion(new Region(600, 200, 80, 30))
            .averageDuration(0.5)
            .build()
    )
    .build();

Configuring Mock Mode

In application.properties:

# Enable mock mode
brobot.mock=true

# Mock timing configuration (faster tests)
brobot.mock.timing.action=0.01
brobot.mock.timing.transition=0.02
brobot.mock.timing.wait=0.04

# Screenshot paths for unit tests
brobot.test.screenshot.path=src/test/resources/screenshots/

Programmatically:

import io.github.jspinak.brobot.config.BrobotProperties;

@Autowired
private BrobotProperties brobotProperties;

@BeforeEach
public void setup() {
    brobotProperties.setMock(true);
}

Complete Integration Test Example

import io.github.jspinak.brobot.test.BrobotTestBase;
import io.github.jspinak.brobot.navigation.StateNavigator;
import io.github.jspinak.brobot.statemanagement.StateMemory;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import static org.junit.jupiter.api.Assertions.*;

@SpringBootTest
public class CompleteWorkflowTest extends BrobotTestBase {

    @Autowired
    private StateNavigator stateNavigator;

    @Autowired
    private StateMemory stateMemory;

    @Test
    public void testCompleteLoginAndNavigationWorkflow() {
        // Arrange: Start at Login screen
        stateMemory.setActiveStates("Login");

        // Act: Navigate through complete workflow
        boolean loginSuccess = stateNavigator.openState("Dashboard");
        boolean settingsSuccess = stateNavigator.openState("Settings");
        boolean profileSuccess = stateNavigator.openState("Profile");

        // Assert: Verify all transitions succeeded
        assertTrue(loginSuccess, "Login should succeed");
        assertTrue(settingsSuccess, "Navigate to Settings should succeed");
        assertTrue(profileSuccess, "Navigate to Profile should succeed");

        // Verify final state
        assertTrue(stateMemory.isActive("Profile"));

        // Verify path was traversed
        assertEquals(3, stateMemory.getTransitionHistory().size());
    }
}

Complete Unit Test Example

import io.github.jspinak.brobot.test.BrobotTestBase;
import io.github.jspinak.brobot.actions.Action;
import io.github.jspinak.brobot.model.state.StateImage;
import io.github.jspinak.brobot.results.ActionResult;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import static org.junit.jupiter.api.Assertions.*;

public class LoginButtonPatternMatchTest extends BrobotTestBase {

    @Autowired
    private Action action;

    @Test
    public void testLoginButtonPatternRecognition() {
        // Arrange: Create StateImage with static screenshot
        StateImage loginButton = new StateImage.Builder()
            .addPatterns("login-button.png")
            .build();

        // Act: Perform pattern matching on fixed screenshot
        // (f_a^unit uses Ξ_x = static screenshot)
        ActionResult result = action.find(loginButton);

        // Assert: Verify pattern matching works
        assertTrue(result.isSuccess(), "Should find login button in screenshot");
        assertNotNull(result.getMatch());
        assertTrue(result.getMatch().getSimilarity() > 0.90,
                   "Similarity should be > 90%");
        assertEquals(450, result.getMatch().getLocation().getX(), 10,
                    "X coordinate should be approximately 450");
        assertEquals(320, result.getMatch().getLocation().getY(), 10,
                    "Y coordinate should be approximately 320");
    }
}

Testing State Definitions

import io.github.jspinak.brobot.test.BrobotTestBase;
import io.github.jspinak.brobot.statemanagement.StateDetector;
import io.github.jspinak.brobot.model.state.State;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import static org.junit.jupiter.api.Assertions.*;

public class StateDefinitionTest extends BrobotTestBase {

    @Autowired
    private StateDetector stateDetector;

    @Autowired
    private State dashboardState;

    @Test
    public void testDashboardStateHasSufficientElements() {
        // Verify state has multiple identifying elements
        assertTrue(dashboardState.getStateImages().size() >= 3,
                  "Dashboard should have at least 3 identifying elements");

        // Verify each element has action history (for mock testing)
        dashboardState.getStateImages().forEach(image -> {
            assertNotNull(image.getActionHistory(),
                         "Each state image should have action history");
        });
    }

    @Test
    public void testDashboardStateDetection() {
        // Arrange: Set mock scene with Dashboard visible
        stateMemory.setActiveStates("Dashboard");

        // Act: Detect which states are active
        Set<String> activeStates = stateDetector.detectActiveStates();

        // Assert: Dashboard should be detected
        assertTrue(activeStates.contains("Dashboard"),
                  "Dashboard state should be detected");
    }
}

Testing Transitions

import io.github.jspinak.brobot.test.BrobotTestBase;
import io.github.jspinak.brobot.actions.methods.basicactions.transitions.TransitionFunction;
import io.github.jspinak.brobot.model.transition.StateTransition;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import static org.junit.jupiter.api.Assertions.*;

public class TransitionTest extends BrobotTestBase {

    @Autowired
    private TransitionFunction transitionFunction;

    @Autowired
    private StateTransition loginToDashboard;

    @Test
    public void testLoginTransitionExecutes() {
        // Arrange: Start at Login state
        stateMemory.setActiveStates("Login");

        // Act: Execute transition
        boolean success = transitionFunction.execute(loginToDashboard);

        // Assert: Transition should succeed
        assertTrue(success, "Login transition should succeed");

        // Verify state changes
        assertFalse(stateMemory.isActive("Login"),
                   "Login state should be deactivated");
        assertTrue(stateMemory.isActive("Dashboard"),
                  "Dashboard state should be activated");
    }
}

Common Patterns for Mock Testing

Pattern 1: Reliable Element (Always Found)

StateImage alwaysFoundButton = new StateImage.Builder()
    .addPatterns("button.png")
    .withActionHistory(
        MockActionHistoryBuilder.builder()
            .successRate(1.0)
            .matchRegion(new Region(400, 300, 100, 50))
            .build()
    )
    .build();

Use Case: Stable UI elements that always appear in the same location.

Pattern 2: Flaky Element (Sometimes Not Found)

StateImage flakyPopup = new StateImage.Builder()
    .addPatterns("popup.png")
    .withActionHistory(
        MockActionHistoryBuilder.builder()
            .successRate(0.6)  // Found 60% of the time
            .matchRegion(new Region(500, 200, 200, 100))
            .build()
    )
    .build();

Use Case: Testing robustness when elements are sometimes missing (popups, loading indicators).

Pattern 3: Multiple Match Locations

StateImage multiLocationElement = new StateImage.Builder()
    .addPatterns("icon.png")
    .withActionHistory(
        MockActionHistoryBuilder.builder()
            .addMatchLocation(100, 50, 0.95)  // Location 1
            .addMatchLocation(200, 50, 0.92)  // Location 2
            .addMatchLocation(300, 50, 0.97)  // Location 3
            .build()
    )
    .build();

Use Case: Elements that can appear in multiple locations (icons in a list, repeated patterns).

Pattern 4: Loading/Dynamic Element

StateImage loadingIndicator = new StateImage.Builder()
    .addPatterns("loading.png")
    .withActionHistory(
        MockActionHistoryBuilder.builder()
            .successRate(0.5)  // Visible for short time
            .averageDuration(2.0)  // Takes longer to find/verify
            .build()
    )
    .build();

Use Case: Elements that appear temporarily (loading spinners, progress bars).

Common Mistakes and How to Avoid Them

Mistake 1: Not Using BrobotTestBase

Wrong:

public class MyTest {
    @Test
    public void testSomething() {
        // Will fail in headless environment
        // No mock mode configured
    }
}

Correct:

public class MyTest extends BrobotTestBase {
    @Test
    public void testSomething() {
        // Mock mode enabled automatically
        // Headless environment supported
    }
}
}

Mistake 2: Testing Without Action History

Wrong:

StateImage button = new StateImage.Builder()
    .addPatterns("button.png")
    .build();
// No action history → mock mode will fail

Correct:

StateImage button = new StateImage.Builder()
    .addPatterns("button.png")
    .withActionHistory(
        MockActionHistoryBuilder.builder()
            .successRate(0.95)
            .matchRegion(new Region(400, 300, 100, 50))
            .build()
    )
    .build();

Mistake 3: Forgetting to Enable Mock Mode

Wrong:

@Test
public void testWithoutMockMode() {
    // Will try to access live GUI → fails in CI/CD
    action.find(stateImage);
}

Correct:

@Test
public void testWithMockMode() {
    // BrobotTestBase enables mock mode automatically
    // OR manually:
    brobotProperties.setMock(true);

    action.find(stateImage);
}

Mistake 4: Not Verifying State Changes

Wrong:

@Test
public void testTransition() {
    transitionFunction.execute(transition);
    // Doesn't verify state actually changed
}

Correct:

@Test
public void testTransition() {
    stateMemory.setActiveStates("Login");

    transitionFunction.execute(loginToDashboard);

    assertFalse(stateMemory.isActive("Login"));
    assertTrue(stateMemory.isActive("Dashboard"));
}

Mistake 5: Using Unrealistic Mock Data

Wrong:

MockActionHistoryBuilder.builder()
    .successRate(1.0)  // 100% success
    .averageDuration(0.0)  // Instant
    .build();
// Doesn't reflect real GUI behavior

Correct:

MockActionHistoryBuilder.builder()
    .successRate(0.92)  // Realistic success rate
    .averageDuration(0.15)  // Realistic timing
    .varianceDuration(0.05)  // Some variance
    .build();
// Reflects actual GUI behavior for better tests

Formal Properties of Testing

Understanding these formal properties helps reason about test behavior:

1. Computational Isomorphism

Property: Mock and live execution produce structurally equivalent results.

f_a(a, Ξ, Θ) produces r_a
f_a^mock(a, S_Ξ) produces r_a^h where r_a^h ≅ r_a

Implication: State Management (M) and Path Traversal (§) cannot distinguish between mock and live results, ensuring tests validate real behavior.

2. Determinism in Unit Tests

Property: Unit tests with fixed scenes produce deterministic results.

f_a^unit(a, Ξ_x) is deterministic
∀ executions: same a, same Ξ_x → same r_a

Implication: Tests are reproducible across environments and runs.

3. Stochasticity Preservation in Integration Tests

Property: Integration tests can preserve stochastic behavior through action history sampling.

If AH contains multiple snapshots for same (o_a, S_Ξ),
selection function can randomly choose → preserves stochasticity

Implication: Integration tests can validate robustness to environmental variability.

4. Model Validation Completeness

Property: Test failures map to specific model deficiencies.

Test fails ⟺ ∃ deficiency in Ω = (E, S, T)
- Missing element: ∃ e ∉ E required for transition
- Missing transition: ∄ t ∈ T connecting s₀ to s_t
- Inadequate state: s ∈ S poorly defined

Implication: Test results directly guide model improvement.

Benefits Summary

Testing automation in model-based GUI automation provides:

Engineering Benefits

• ✅ Standard software practices: Unit tests, integration tests, TDD
• ✅ CI/CD integration: Run tests without GUI in automated pipelines
• ✅ Fast feedback loops: Tests run in milliseconds, not minutes
• ✅ Reproducible results: Eliminate environmental variability

Model Quality Benefits

• ✅ Validate State Structure: Tests verify Ω = (E, S, T) accuracy
• ✅ Identify gaps: Missing elements, states, or transitions
• ✅ Iterative refinement: Tests guide model improvement
• ✅ Regression detection: Catch model degradation

Development Benefits

• ✅ Confidence: Know automation works before deployment
• ✅ Refactoring safety: Tests catch breaking changes
• ✅ Documentation: Tests demonstrate intended behavior
• ✅ Debugging: Isolate issues to specific components

Implementation Files in Brobot

Key Classes:

1. BrobotTestBase (test base class)

   • Location: /library/src/test/java/io/github/jspinak/brobot/test/BrobotTestBase.java
   • Provides: Mock mode setup, headless support, fast timings

2. ActionRecord (Action Snapshot implementation)

   • Location: /library/src/main/java/io/github/jspinak/brobot/actions/actionHistory/ActionRecord.java
   • Implements: AS = (o_a^h, S_Ξ^h, r_a^h)

3. MockActionHistoryBuilder (mock data builder)

   • Location: /library/src/main/java/io/github/jspinak/brobot/mock/MockActionHistoryBuilder.java
   • Provides: Fluent API for creating action histories

4. BrobotProperties (configuration)

   • Location: /library/src/main/java/io/github/jspinak/brobot/config/BrobotProperties.java
   • Controls: Mock mode, timings, paths

Further Reading

Academic Foundation

• Academic Foundation - Research background and citations
  • Section 3.4: Testing Automation vs Automated Testing
  • Section 11: Testing Automation (Integration, Unit, Model Validation)
  • Table 3: Comparison of Integration and Unit Tests

Related Models

• Overall Model - Complete formal model (Ξ, Ω, a, M, τ, §)
• States - State Structure and state management
• Transitions - Transition model and path traversal

Practical Testing Guides

• Mock Mode Guide - Detailed mock mode configuration
• Unit Testing Guide - Unit test patterns
• Integration Testing Guide - Integration test patterns
• Test Utilities - Helper classes and utilities

Implementation

• AI Brobot Project Creation Guide - Complete API reference
• Getting Started - Hands-on tutorials

Appendix: Mathematical Notation Quick Reference

• f_a^mock = Mock action function (uses historical data)
• f_a^unit = Unit test action function (uses fixed scenes)
• AH = Action History (set of recorded snapshots)
• AS = Action Snapshot (single recorded action)
• o_a^h = Historical action parameters
• S_Ξ^h = Historical active states
• r_a^h = Historical action results
• ≅ = Structural equivalence
• Ξ_x = Fixed scene (screenshot) for unit testing
• Ω = State Structure (E, S, T) being validated through testing