Version: Latest

Declarative Region Definition

Brobot 1.1.0 introduces a powerful declarative approach to defining search regions for StateImages. This guide explains how to define regions that are dynamically calculated relative to other state objects.

Overview

The declarative approach allows you to:

Define search regions relative to other state objects
Apply adjustments and fixed dimensions
Eliminate manual region calculations in your action code
Create more maintainable and reusable state definitions

SearchRegionOnObject

The SearchRegionOnObject class enables dynamic region definition:

public class SearchRegionOnObject {
    private StateObject.Type targetType;     // Type of target object (IMAGE, REGION, etc.)
    private String targetStateName;          // Name of the state containing the target
    private String targetObjectName;         // Name of the specific object
    private MatchAdjustmentOptions adjustments; // Position and size adjustments (reuses existing class)
}

The adjustments field uses the standard MatchAdjustmentOptions class for consistency with other Brobot operations.

Basic Usage

Simple Relative Region

Define a search region relative to another StateImage:

StateImage searchArea = new StateImage.Builder()
    .addPatterns("search-icon.png")
    .setName("SearchIcon")
    .setSearchRegionOnObject(SearchRegionOnObject.builder()
        .targetType(StateObject.Type.IMAGE)
        .targetStateName("MainMenu")
        .targetObjectName("MenuButton")
        .build())
    .build();

With Adjustments

Apply position and size adjustments to the derived region:

StateImage icon = new StateImage.Builder()
    .addPatterns("status-icon.png")
    .setName("StatusIcon")
    .setSearchRegionOnObject(SearchRegionOnObject.builder()
        .targetType(StateObject.Type.IMAGE)
        .targetStateName("Dashboard")
        .targetObjectName("HeaderBar")
        .adjustments(MatchAdjustmentOptions.builder()
            .addX(10)    // Move 10 pixels right
            .addY(-5)    // Move 5 pixels up
            .addW(50)    // Expand width by 50 pixels
            .addH(20)    // Expand height by 20 pixels
            .build())
        .build())
    .build();

With Fixed Dimensions

Override the calculated dimensions with fixed values:

StateImage button = new StateImage.Builder()
    .addPatterns("submit-button.png")
    .setName("SubmitButton")
    .setSearchRegionOnObject(SearchRegionOnObject.builder()
        .targetType(StateObject.Type.IMAGE)
        .targetStateName("Form")
        .targetObjectName("FormTitle")
        .adjustments(MatchAdjustmentOptions.builder()
            .addY(100)        // Move down 100 pixels
            .absoluteW(200)   // Fixed width of 200 pixels
            .absoluteH(50)    // Fixed height of 50 pixels
            .build())
        .build())
    .build();

Real-World Example: Claude Automator

The claude-automator project demonstrates this pattern effectively:

@State
@Getter
public class WorkingState {
    
    private final StateImage claudeIcon;
    
    public WorkingState() {
        // The search region will be dynamically defined relative to the prompt
        claudeIcon = new StateImage.Builder()
            .addPatterns("working/claude-icon-1", 
                        "working/claude-icon-2", 
                        "working/claude-icon-3", 
                        "working/claude-icon-4")
            .setName("ClaudeIcon")
            .setSearchRegionOnObject(SearchRegionOnObject.builder()
                    .targetType(StateObject.Type.IMAGE)
                    .targetStateName("Prompt")
                    .targetObjectName("ClaudePrompt")
                    .adjustments(MatchAdjustmentOptions.builder()
                            .addX(3)      // Slight offset to the right
                            .addY(10)     // Below the prompt
                            .addW(30)     // Wider search area
                            .addH(55)     // Taller search area
                            .build())
                    .build())
            .build();
    }
}

This declarative approach eliminates the need for manual region calculations:

// Before: Manual calculation in action code
private void setupIconRegion(Region promptRegion) {
    Region iconRegion = new Region(promptRegion);
    iconRegion.adjust(3, 10, 30, 55);
    workingState.getClaudeIcon().setSearchRegions(iconRegion);
}

// After: Automatic calculation based on declaration
// No manual setup needed - just use the StateImage directly
ActionResult result = action.perform(findOptions, workingState.getClaudeIcon());

Builder Methods

The SearchRegionOnObject.builder() provides a fluent API for configuration:

Basic Structure

SearchRegionOnObject.builder()
    .targetType(StateObject.Type.IMAGE)  // Required: Type of target
    .targetStateName("StateName")        // Required: State containing target
    .targetObjectName("ObjectName")      // Required: Name of target object
    .adjustments(...)                    // Optional: Position/size adjustments using MatchAdjustmentOptions
    .build()

MatchAdjustmentOptions Builder

.adjustments(MatchAdjustmentOptions.builder()
    .addX(10)         // Add to x position
    .addY(20)         // Add to y position  
    .addW(30)         // Add to width
    .addH(40)         // Add to height
    .absoluteW(200)   // Override with fixed width (optional)
    .absoluteH(100)   // Override with fixed height (optional)
    .targetPosition(Position.CENTER)  // Target position within region (optional)
    .targetOffset(new Location(5, 5)) // Additional offset (optional)
    .build())

Key Differences from Standard Match Adjustments

When used with SearchRegionOnObject, only position and dimension adjustments apply
targetPosition and targetOffset are ignored for search region calculation
Use negative values in addX/addY to move left/up
Use absoluteW/absoluteH set to -1 (default) to not override dimensions

Cross-State References

SearchRegionOnObject supports referencing objects from different states:

// In LoginState
StateImage loginButton = new StateImage.Builder()
    .addPatterns("login-button.png")
    .setName("LoginButton")
    .build();

// In DashboardState - reference login button location
StateImage notification = new StateImage.Builder()
    .addPatterns("notification.png")
    .setName("Notification")
    .setSearchRegionOnObject(new SearchRegionOnObject.Builder()
        .targetType(StateObject.Type.IMAGE)
        .targetStateName("Login")  // Different state
        .targetObjectName("LoginButton")
        .yAdjust(-50)  // Above the login button
        .build())
    .build();

How Cross-State Dependencies Work

When you define a cross-state dependency:

Registration Phase: When states are loaded, the SearchRegionDependencyInitializer automatically registers all dependencies with the DynamicRegionResolver.
Runtime Resolution: When a FIND operation succeeds:
- The FindPipeline calls updateDependentSearchRegions()
- All objects depending on the found object have their search regions updated
- The updates apply the configured adjustments
Automatic Updates: Search regions are dynamically updated each time the target object is found in a new location.

Example flow:

// 1. ClaudePrompt is found at location (100, 200)
// 2. ClaudeIcon's search region is automatically updated to (103, 210, width+30, height+55)
// 3. Next search for ClaudeIcon uses this updated region

Integration with State-Aware Scheduling

The declarative approach works seamlessly with StateAwareScheduler:

@Service
public class MonitoringService {
    
    private final StateAwareScheduler scheduler;
    private final WorkingState workingState;
    
    public void startMonitoring() {
        // Configure state checking
        StateCheckConfiguration config = new StateCheckConfiguration.Builder()
            .withRequiredStates(List.of("Prompt", "Working"))
            .build();
        
        // Schedule monitoring - regions are resolved automatically
        scheduler.scheduleWithStateCheck(
            executor,
            this::checkIcon,
            config,
            5, 2, TimeUnit.SECONDS
        );
    }
    
    private void checkIcon() {
        // The search region is automatically calculated based on
        // the current location of ClaudePrompt in PromptState
        ActionResult result = action.perform(
            new PatternFindOptions.Builder().build(),
            workingState.getClaudeIcon()
        );
    }
}

Best Practices

Use Descriptive Names: Give clear names to both source and target objects
```
.targetObject("HeaderNavigationBar")  // Clear and specific
```
Document Adjustments: Comment on why specific adjustments are used
```
.adjustments(0, 50, 0, 0)  // Below header, same width
```
Consider State Dependencies: Ensure target states are loaded when needed
```
.withRequiredStates(List.of("SourceState", "TargetState"))
```

Use Fixed Dimensions Sparingly: Prefer relative sizing for responsiveness

// Good: Relative adjustment
.wAdjust(20)  // Slightly wider than source

// Use fixed only when necessary
.width(100)   // Fixed width for consistent button size

Migration from Manual Approach

To migrate existing code:

Identify Manual Region Calculations

// Old approach
Region baseRegion = findResult.getRegion();
Region searchRegion = new Region(
    baseRegion.x() + 10,
    baseRegion.y() + 50,
    baseRegion.w() + 20,
    baseRegion.h()
);
stateImage.setSearchRegions(searchRegion);

Convert to Declarative Definition

// New approach
stateImage = new StateImage.Builder()
    .addPatterns("pattern.png")
    .setSearchRegionOnObject(SearchRegionOnObject.builder()
        .targetType(StateObject.Type.IMAGE)
        .targetStateName("Base")  // @State removes "State" suffix from class name
        .targetObjectName("BaseImage")
        .adjustments(MatchAdjustmentOptions.builder()
            .addX(10)
            .addY(50)
            .addW(20)
            .addH(0)
            .build())
        .build())
    .build();

Remove Manual Region Management
- Delete region calculation code
- Remove region storage variables
- Simplify action methods

Implementation Architecture

The declarative region system consists of several key components:

Core Components

SearchRegionOnObject: The configuration object that defines the dependency
- Holds target state/object information
- Contains adjustment and dimension settings
- Attached to StateImages during state construction
SearchRegionDependencyRegistry: Tracks all dependencies
- Maps source objects to their dependents
- Provides lookup for dependent objects when sources are found
- Thread-safe for concurrent access
DynamicRegionResolver: Resolves and updates regions
- Calculates actual regions based on found objects
- Updates dependent object search regions
- Handles both same-state and cross-state dependencies
SearchRegionDependencyInitializer: Initializes the system
- Listens for StatesRegisteredEvent
- Collects all StateObjects with dependencies
- Registers them with the DynamicRegionResolver
FindPipeline Integration: Triggers updates
- Calls updateDependentSearchRegions() after successful finds
- Ensures dependent regions are updated before next search

Initialization Flow

Application Start
    ↓
States Loaded (@State classes instantiated)
    ↓
StatesRegisteredEvent Published
    ↓
SearchRegionDependencyInitializer Receives Event
    ↓
Collects All StateObjects with SearchRegionOnObject
    ↓
Registers Dependencies with DynamicRegionResolver
    ↓
System Ready for Dynamic Region Updates

Runtime Flow

FIND Operation Executes
    ↓
Matches Found
    ↓
FindPipeline.updateDependentSearchRegions()
    ↓
For Each Match:
    - Get Dependent Objects from Registry
    - Calculate New Search Region
    - Update Dependent Object's Search Region
    ↓
Next FIND Uses Updated Regions

Troubleshooting

Region Not Found

Verify target state and object names match exactly
Ensure target state is active when searching
Check that target object has been found at least once
Enable logging: logging.level.io.github.jspinak.brobot.action.internal.region=DEBUG

Incorrect Region Position

Log the resolved region for debugging:

ActionResult result = action.perform(findOptions, stateImage);
log.info("Search region: {}", result.getSearchedRegion());

Adjust the adjustment values incrementally

Consider using visual feedback:

action.perform(new HighlightOptions.Builder().build(), stateImage);

Dependencies Not Working

Verify SearchRegionDependencyInitializer is being instantiated
Check logs for "Registered search region dependency" messages
Ensure Spring component scanning includes brobot packages
Verify target object names match exactly (case-sensitive)

Performance Considerations

Region resolution happens on each search
Dependencies are registered once at startup
Updates only occur when source objects are found
Consider using fixed regions for static layouts

Summary

Declarative region definition in Brobot 1.1.0 provides:

Cleaner, more maintainable code
Dynamic adaptation to UI changes
Better separation of concerns
Seamless integration with state management

By defining regions declaratively, you create more robust automation that adapts to UI variations while keeping your action code focused on business logic rather than region calculations.

Overview​

SearchRegionOnObject​

Basic Usage​

Simple Relative Region​

With Adjustments​

With Fixed Dimensions​

Real-World Example: Claude Automator​

Builder Methods​

Basic Structure​

MatchAdjustmentOptions Builder​

Key Differences from Standard Match Adjustments​

Cross-State References​

How Cross-State Dependencies Work​

Integration with State-Aware Scheduling​

Best Practices​

Migration from Manual Approach​

Implementation Architecture​

Core Components​

Initialization Flow​

Runtime Flow​

Troubleshooting​

Region Not Found​

Incorrect Region Position​

Dependencies Not Working​

Performance Considerations​

Summary​