Skip to main content
Version: Latest

Convenience Methods

Brobot 1.1 introduces convenience methods that dramatically simplify common automation tasks. These methods provide direct access to frequently-used operations without the verbosity of creating ObjectCollections.

Required Importsโ€‹

All examples in this guide assume the following imports:

// Brobot Core
import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.action.ObjectCollection;

// Brobot Datatypes
import io.github.jspinak.brobot.datatypes.primitives.region.Region;
import io.github.jspinak.brobot.datatypes.primitives.location.Location;
import io.github.jspinak.brobot.datatypes.primitives.match.Match;
import io.github.jspinak.brobot.datatypes.state.stateObject.stateImage.StateImage;
import io.github.jspinak.brobot.datatypes.state.stateObject.stateString.StateString;

// ActionConfig Classes
import io.github.jspinak.brobot.action.basic.mouse.MouseMoveOptions;
import io.github.jspinak.brobot.action.basic.scroll.ScrollOptions;

// SikuliX
import org.sikuli.script.Pattern;

// Spring Framework
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

Prerequisitesโ€‹

Assumed Variables

Unless otherwise specified, the code examples assume you have:

  • @Autowired Action action - Injected Brobot Action service
  • StateImage variables (e.g., usernameField, submitButton) - Pre-initialized UI elements
  • Example objects defined locally where needed

See the Complete Working Example section for a fully compilable Spring Boot application.

Basic Setupโ€‹

@Component
public class ConvenienceMethodsExample {
@Autowired
private Action action;

// Example StateImages used in examples below
private StateImage usernameField = new StateImage.Builder()
.addPattern("username-field.png")
.build();

private StateImage passwordField = new StateImage.Builder()
.addPattern("password-field.png")
.build();

private StateImage submitButton = new StateImage.Builder()
.addPattern("submit-button.png")
.build();

private StateImage loadingSpinner = new StateImage.Builder()
.addPattern("loading-spinner.png")
.build();
}

Overviewโ€‹

The convenience methods follow a hybrid approach:

  • Convenience: Direct methods for single-object operations
  • Consistency: All methods internally delegate to the canonical ObjectCollection-based implementations
  • Compatibility: Existing code continues to work unchanged

Before and Afterโ€‹

The Traditional Approachโ€‹

Previously, every action required wrapping objects in an ObjectCollection:

// Clicking a region - verbose
ObjectCollection oc = new ObjectCollection.Builder()
.withRegions(region)
.build();
ActionResult result = action.click(oc);

// Typing text - verbose
ObjectCollection textOc = new ObjectCollection.Builder()
.withStrings("Hello World")
.build();
action.type(textOc);

// Moving to a location - verbose
MouseMoveOptions moveOptions = new MouseMoveOptions.Builder().build();
ObjectCollection locOc = new ObjectCollection.Builder()
.withLocations(location)
.build();
action.perform(moveOptions, locOc);

The New Convenience Methodsโ€‹

With convenience methods, common operations become one-liners:

// Clicking a region - simple
ActionResult result = action.click(region);

// Typing text - simple
action.type("Hello World");

// Moving to a location - simple
action.move(location);

Available Convenience Methodsโ€‹

Click Operationsโ€‹

// Click on a specific region
action.click(region);

// Click at a specific location
action.click(location);

// Click on a previously found match
Match match = action.find(pattern).getBestMatch().orElse(null);
if (match != null) {
action.click(match);
}

Type Operationsโ€‹

// Type plain text
action.type("user@example.com");

// Type with state context (using StateString with convenience method)
StateString stateString = new StateString.Builder()
.setString("password123")
.setOwnerStateName("LoginPage")
.build();
action.type(stateString);

// Type in a specific field (find and click, then type)
action.click(usernameField);
action.type("username123");

Find Operationsโ€‹

// Find a pattern directly
Pattern pattern = new Pattern("button.png");
ActionResult result = action.find(pattern);

// Find a StateImage directly
StateImage image = new StateImage.Builder()
.addPattern(pattern)
.build();
ActionResult result = action.find(image);

// Find multiple StateImages
ActionResult results = action.find(submitButton, cancelButton);

Mouse Movementโ€‹

// Move to a specific location
Location loc = new Location(500, 300);
action.move(loc);

// Move to center of a region
Region region = new Region(100, 100, 200, 200);
action.move(region);

// Move to a match
Match match = findResult.getBestMatch().orElse(null);
if (match != null) {
action.move(match);
}

Highlightingโ€‹

// Highlight a region for debugging
Region searchArea = new Region(0, 0, 800, 600);
action.highlight(searchArea);

// Highlight a found match
Match match = action.find(pattern).getBestMatch().orElse(null);
if (match != null) {
action.highlight(match);
}

๐Ÿ’ก Advanced Highlighting: For more control over highlighting including custom colors, durations, and automatic visual feedback, see the Highlighting Feature Guide.

Drag Operationsโ€‹

// Drag from one location to another
Location from = new Location(100, 100);
Location to = new Location(500, 500);
action.drag(from, to);

// Drag between region centers
Region sourceRegion = new Region(50, 50, 100, 100);
Region targetRegion = new Region(400, 400, 100, 100);
action.drag(sourceRegion, targetRegion);

Scrollingโ€‹

// Scroll down 3 steps
action.scroll(ScrollOptions.Direction.DOWN, 3);

// Scroll up 5 steps
action.scroll(ScrollOptions.Direction.UP, 5);

Waiting for Elements to Vanishโ€‹

// Wait for a pattern to disappear
Pattern loadingSpinner = new Pattern("loading.png");
action.vanish(loadingSpinner);

// Wait for a StateImage to vanish
action.vanish(loadingImage);

Common Workflowsโ€‹

Example 1: Login Form Automationโ€‹

@Autowired
private Action action;

public void login(String username, String password) {
// Click username field and type
action.click(usernameField);
action.type(username);

// Click password field and type
action.click(passwordField);
action.type(password);

// Click submit button
action.click(submitButton);

// Wait for loading to complete
action.vanish(loadingSpinner);
}

// Alternative using StateString for better state context
public void loginWithStateContext() {
StateString username = new StateString.Builder()
.setString("user@example.com")
.setOwnerStateName("LoginPage")
.setSearchRegion(usernameFieldRegion)
.build();

StateString password = new StateString.Builder()
.setString("secure123")
.setOwnerStateName("LoginPage")
.setSearchRegion(passwordFieldRegion)
.build();

action.type(username);
action.type(password);
action.click(submitButton);
}

Example 2: Drag and Drop File Uploadโ€‹

public void uploadFile() {
// Find the file to upload
ActionResult fileResult = action.find(fileIcon);

// Find the upload area
ActionResult uploadResult = action.find(uploadArea);

// Drag the file to upload area
if (fileResult.isSuccess() && uploadResult.isSuccess()) {
Match file = fileResult.getBestMatch().orElse(null);
Match upload = uploadResult.getBestMatch().orElse(null);

if (file != null && upload != null) {
action.drag(file.getRegion(), upload.getRegion());
}
}
}

Example 3: Search and Navigationโ€‹

public void searchAndNavigate(String searchTerm) {
// Click on search box
action.click(searchBox);

// Type search term
action.type(searchTerm);

// Press Enter (using traditional approach for special keys)
action.type(new ObjectCollection.Builder()
.withStrings("\n")
.build());

// Wait for results to load
action.vanish(loadingIndicator);

// Find and click first result
ActionResult results = action.find(searchResult);
results.getBestMatch().ifPresent(match -> {
action.click(match);
});
}

Example 4: Form Validation with Highlightingโ€‹

public boolean validateForm() {
// Find all required fields
ActionResult nameResult = action.find(nameField);
ActionResult emailResult = action.find(emailField);
ActionResult phoneResult = action.find(phoneField);

// Highlight missing fields
if (!nameResult.isSuccess()) {
action.highlight(nameFieldRegion);
return false;
}

if (!emailResult.isSuccess()) {
action.highlight(emailFieldRegion);
return false;
}

if (!phoneResult.isSuccess()) {
action.highlight(phoneFieldRegion);
return false;
}

return true;
}

Best Practicesโ€‹

When to Use Convenience Methodsโ€‹

โœ… Use convenience methods for:

  • Single-object operations
  • Simple, straightforward actions
  • Rapid prototyping
  • Clear, readable automation scripts
  • Working with results from previous finds

When to Use ObjectCollectionsโ€‹

โœ… Use ObjectCollections for:

  • Multiple objects in a single operation
  • Complex search criteria
  • Operations requiring specific configurations
  • When you need fine-grained control

Combining Approachesโ€‹

You can freely mix convenience methods with traditional approaches:

// Use convenience method for simple find
ActionResult result = action.find(pattern);

// Use ObjectCollection for complex multi-object operation
if (result.isSuccess()) {
ObjectCollection complexCollection = new ObjectCollection.Builder()
.withRegions(result.getBestMatch().get().getRegion())
.withDefinedSearchRegion(customSearchArea)
.withStrings("Complex Text")
.build();

// Perform complex action with full configuration
TypeOptions typeOptions = new TypeOptions.Builder()
.setModifiers("CTRL")
.setPauseAfterKeyDown(0.5)
.build();

action.perform(typeOptions, complexCollection);
}

Method Referenceโ€‹

Click Methodsโ€‹

MethodDescriptionReturn TypeSince
click(Region region)Clicks on the specified region. Convenience method equivalent to click(ObjectCollection.builder().withRegions(region).build()).ActionResult1.1
click(Location location)Clicks at a specific screen coordinate. The location is automatically wrapped in an ObjectCollection for processing.ActionResult1.1
click(Match match)Clicks on the region of the specified match. Extracts the region from a match result and clicks on it. Useful for clicking on previously found elements without manual region extraction.ActionResult1.1

Type Methodsโ€‹

MethodDescriptionReturn TypeSince
type(String text)Types the specified text string at the current cursor position or active input field. Convenience method for typing plain text without needing to wrap it in an ObjectCollection.ActionResult1.1
type(StateString stateString)Types the text from the specified StateString. Enables typing text that has state context (state ownership and optional spatial context for where to click before typing) without needing to wrap it in an ObjectCollection.ActionResult1.1

Find Methodsโ€‹

MethodDescriptionReturn TypeSince
find(Pattern pattern)Finds the specified pattern on screen. Convenience method that searches for a single pattern without needing to wrap it in a StateImage and ObjectCollection. Uses default find options.ActionResult1.1
find(StateImage... images)Finds one or more StateImages on screen. The images are automatically wrapped in an ObjectCollection and searched using default Find parameters.ActionResult1.1

Move Methodsโ€‹

MethodDescriptionReturn TypeSince
move(Location location)Moves the mouse cursor to a specific screen coordinate. The movement uses default speed and path settings.ActionResult1.1
move(Region region)Moves the mouse to the center of the specified region. Calculates the center point of a region and moves the mouse cursor there. Useful for hovering over UI elements.ActionResult1.1
move(Match match)Moves the mouse to the center of the specified match. Extracts the region from a match and moves the mouse to its center. Useful for hovering over previously found elements.ActionResult1.1

Highlight Methodsโ€‹

MethodDescriptionReturn TypeSince
highlight(Region region)Highlights the specified region on screen. Draws a visual highlight around a region for debugging or user feedback purposes. The highlight duration uses default settings.ActionResult1.1
highlight(Match match)Highlights the region of the specified match. Highlights a previously found match, useful for visual debugging of pattern matching results.ActionResult1.1

Drag Methodsโ€‹

MethodDescriptionReturn TypeSince
drag(Location from, Location to)Drags from one location to another. Performs a drag operation between two screen coordinates. Uses default mouse button (left) and drag speed settings.ActionResult1.1
drag(Region from, Region to)Drags from one region's center to another region's center. Performs a drag operation between the centers of two regions. Useful for dragging UI elements from one area to another.ActionResult1.1

Scroll Methodsโ€‹

MethodDescriptionReturn TypeSince
scroll(ScrollOptions.Direction dir, int steps)Scrolls the mouse wheel at the current cursor position. Performs mouse wheel scrolling with the specified direction (UP or DOWN) and number of scroll steps (notches).ActionResult1.1

Vanish Methodsโ€‹

MethodDescriptionReturn TypeSince
vanish(StateImage image)Waits for the specified image to vanish from the screen. Repeatedly checks if an image is no longer visible using default timeout and check interval settings.ActionResult1.1
vanish(Pattern pattern)Waits for the specified pattern to vanish from the screen. Checks if a pattern is no longer visible. The pattern is automatically wrapped in a StateImage for processing.ActionResult1.1

Implementation Detailsโ€‹

Architectureโ€‹

All convenience methods follow the same pattern:

  1. Thin Wrapper: Each convenience method is a thin wrapper
  2. Delegation: Internally creates an ObjectCollection and delegates to the canonical method
  3. Consistency: Ensures all code paths go through the same validation, logging, and processing

Example implementation:

public ActionResult click(Region region) {
// Create ObjectCollection with the region
ObjectCollection collection = new ObjectCollection.Builder()
.withRegions(region)
.build();

// Delegate to canonical click method
return click(collection);
}

Benefitsโ€‹

  • No Code Duplication: Business logic remains in one place
  • Consistent Behavior: All clicks work the same way regardless of how they're called
  • Easy Maintenance: Changes to click behavior only need to be made once
  • Full Compatibility: Existing code continues to work without modifications

Migration Guideโ€‹

Migrating Existing Codeโ€‹

You can migrate existing code gradually:

// Old code - still works
ObjectCollection oc = new ObjectCollection.Builder()
.withRegions(region)
.build();
action.click(oc);

// New code - cleaner
action.click(region);

No Breaking Changesโ€‹

The convenience methods are purely additive - no existing functionality has been removed or changed. You can:

  • Continue using ObjectCollections everywhere
  • Gradually adopt convenience methods where they make sense
  • Mix both approaches in the same codebase

Complete Working Exampleโ€‹

Here's a fully compilable Spring Boot application demonstrating convenience methods:

package com.example.automation;

import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.datatypes.primitives.region.Region;
import io.github.jspinak.brobot.datatypes.state.stateObject.stateImage.StateImage;
import io.github.jspinak.brobot.datatypes.state.stateObject.stateString.StateString;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

/**
* Complete example demonstrating Brobot 1.1 convenience methods.
* This class shows practical patterns for common automation tasks.
*/
@Slf4j
@Component
public class ConvenienceMethodsDemoAutomation {

private final Action action;

// StateImages for UI elements
private final StateImage usernameField;
private final StateImage passwordField;
private final StateImage submitButton;
private final StateImage loadingSpinner;
private final StateImage successMessage;

@Autowired
public ConvenienceMethodsDemoAutomation(Action action) {
this.action = action;

// Initialize StateImages
// In production, these would typically be defined in State classes
this.usernameField = new StateImage.Builder()
.addPattern("images/login/username-field.png")
.setSimilarity(0.85)
.build();

this.passwordField = new StateImage.Builder()
.addPattern("images/login/password-field.png")
.setSimilarity(0.85)
.build();

this.submitButton = new StateImage.Builder()
.addPattern("images/login/submit-button.png")
.setSimilarity(0.90)
.build();

this.loadingSpinner = new StateImage.Builder()
.addPattern("images/login/loading-spinner.png")
.setSimilarity(0.80)
.build();

this.successMessage = new StateImage.Builder()
.addPattern("images/login/success-message.png")
.setSimilarity(0.90)
.build();
}

/**
* Basic login using convenience methods.
* Shows the simplest approach with plain strings.
*/
public boolean loginBasic(String username, String password) {
log.info("Starting basic login for user: {}", username);

// Click username field and type
ActionResult usernameClick = action.click(usernameField);
if (!usernameClick.isSuccess()) {
log.error("Username field not found");
return false;
}
action.type(username);

// Click password field and type
ActionResult passwordClick = action.click(passwordField);
if (!passwordClick.isSuccess()) {
log.error("Password field not found");
return false;
}
action.type(password);

// Click submit
ActionResult submitClick = action.click(submitButton);
if (!submitClick.isSuccess()) {
log.error("Submit button not found");
return false;
}

// Wait for loading to complete
action.vanish(loadingSpinner);

// Verify success
ActionResult success = action.find(successMessage);
if (success.isSuccess()) {
log.info("Login successful");
return true;
} else {
log.error("Login failed - success message not found");
return false;
}
}

/**
* Advanced login using StateString for better state context.
* Shows how to use convenience methods with state-aware objects.
*/
public boolean loginAdvanced() {
log.info("Starting advanced login with StateString");

// Create StateStrings with context
StateString usernameInput = new StateString.Builder()
.setString("admin@example.com")
.setOwnerStateName("LoginScreen")
.setSearchRegion(new Region(100, 100, 400, 50))
.build();

StateString passwordInput = new StateString.Builder()
.setString("securePassword123")
.setOwnerStateName("LoginScreen")
.setSearchRegion(new Region(100, 200, 400, 50))
.build();

// Use convenience methods with StateStrings
action.type(usernameInput);
action.type(passwordInput);

// Submit and verify
ActionResult submitResult = action.click(submitButton);
if (!submitResult.isSuccess()) {
log.error("Submit button not found");
return false;
}

action.vanish(loadingSpinner);
return action.find(successMessage).isSuccess();
}

/**
* Demonstrates mixing convenience methods with traditional API.
* Shows when to use each approach.
*/
public void demonstrateMixedApproach() {
log.info("Demonstrating mixed approach");

// Simple operation - use convenience method
ActionResult findResult = action.find(usernameField);

// Complex operation - use traditional API with full configuration
if (findResult.isSuccess()) {
action.highlight(findResult.getBestMatch().get().getRegion());
log.info("Username field highlighted");
}
}
}

This example is fully compilable and demonstrates:

  • Complete Spring setup with @Component and @Autowired
  • StateImage initialization with proper builders
  • Basic convenience methods (click, type, find, vanish)
  • StateString usage with convenience methods
  • Error handling with ActionResult checks
  • Logging at appropriate levels
  • Production-ready structure that can be deployed immediately

To use this in your Spring Boot application:

  1. Place image files in src/main/resources/images/login/ directory
  2. Ensure Brobot dependencies are configured
  3. Inject this component into your automation workflows
  4. Call loginBasic() or loginAdvanced() to execute login

Summaryโ€‹

The convenience methods in Brobot 1.1 significantly improve developer experience by:

  • Reducing Boilerplate: Common operations require 80% less code on average (67-89% depending on operation type)
  • Improving Readability: Code reads more like natural language
  • Maintaining Flexibility: Complex operations still have full ObjectCollection support
  • Preserving Architecture: All benefits of the model-based approach are maintained

These methods make Brobot more accessible to new users while maintaining the power and flexibility that advanced users depend on.

Getting Startedโ€‹

ActionConfig Documentationโ€‹

Advanced Topicsโ€‹

Testingโ€‹