Skip to main content
Version: Latest

Fail-Safe Image Loading Strategy

Overview​

Brobot implements a fail-safe approach to image loading that ensures automation continues even when images are missing or cannot be loaded. This strategy prioritizes robustness and continuous execution over strict validation.

Core Philosophy​

Missing images should never break automation execution. Instead, they should:

  • Log clear error messages for debugging
  • Result in failed pattern matches (not exceptions)
  • Allow the automation to continue and potentially recover

This approach is essential for GUI automation where:

  • Images might be temporarily unavailable during development
  • Network issues might prevent loading remote images
  • File system permissions might block access
  • Images might be intentionally removed or renamed

Implementation Details​

StateImage Pattern Loading Behavior​

When creating a StateImage with pattern images:

import io.github.jspinak.brobot.model.state.StateImage;

StateImage button = new StateImage.Builder()
    .addPatterns("button.png")
    .setName("Button")
    .build();

The pattern loading follows these rules:

  1. In Mock Mode: Image loading is skipped entirely

    • Internal Pattern objects have null images
    • Pattern names are extracted from filenames
    • No file system access occurs
    • StateImage is created successfully

  2. In Live Mode - Image Found:

    • Images are loaded into memory
    • Patterns are fully functional
    • Debug log confirms successful loading
    • StateImage contains valid Pattern objects

  3. In Live Mode - Image Missing:

    • No exception is thrown
    • Error is logged: "Failed to load image: button.png. Pattern will have null image and find operations will fail."
    • Internal Pattern objects have null images
    • StateImage is still created successfully
    • Find operations using this StateImage will fail gracefully

Spring Initialization Handling​

During Spring context initialization, image loading is automatically deferred to avoid unnecessary failures. Internally, Pattern objects detect when Spring is initializing and defer image loading:

// Internal Pattern implementation (for reference)
if (isSpringContextInitializing()) {
    // Defer loading until Spring context is ready
    this.needsDelayedLoading = true;
    this.image = null;
    return;
}

Images are loaded lazily on first use after Spring initialization completes. This is handled automatically by Brobot - no user action required.

Error Handling Strategy​

What Happens When Images Are Missing​

  1. Pattern Creation: Succeeds with null image
  2. Find Operations: Return empty/failed results
  3. Click/Type Actions: Fail gracefully with appropriate ActionResult
  4. State Transitions: May fail if dependent on missing images
  5. Automation Flow: Continues, allowing for recovery logic

Example Recovery Pattern​

import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.model.state.StateImage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

// StateImage with potentially missing image
StateImage submitButton = new StateImage.Builder()
    .addPatterns("submit.png")
    .setName("Submit Button")
    .build();

// Find operation handles missing images gracefully
ActionResult result = action.find(submitButton);

if (!result.isSuccess()) {
    // Recovery logic
    Logger logger = LoggerFactory.getLogger(getClass());
    logger.warn("Submit button not found, trying alternative approach");

    // Try alternative pattern
    StateImage alternativeSubmit = new StateImage.Builder()
        .addPatterns("submit-alt.png")
        .setName("Alternative Submit")
        .build();
    result = action.find(alternativeSubmit);
}

Benefits of This Approach​

1. Development Flexibility​

  • Start writing automation before all images are captured
  • Test automation logic without complete image sets
  • Iterate quickly without image dependency blocks

2. Production Resilience​

  • Temporary file system issues don't crash automation
  • Network failures for remote images are handled gracefully
  • Partial image sets still allow partial functionality

3. Testing Advantages​

  • Mock mode tests run without any image files
  • Unit tests don't require image resources
  • CI/CD pipelines work without image assets

4. Clear Debugging​

  • Error logs pinpoint exactly which images failed to load
  • Automation continues, showing full execution path
  • Easy to identify and fix missing image issues

Best Practices​

1. Always Check ActionResults​

import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.model.state.StateImage;

StateImage button = new StateImage.Builder()
    .addPatterns("button.png")
    .build();

ActionResult result = action.click(button);
if (!result.isSuccess()) {
    // Handle the failure appropriately
}

2. Provide Fallback Patterns​

StateImage supports multiple patterns natively - Brobot automatically tries each pattern until one succeeds:

import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.model.state.StateImage;

// StateImage with multiple pattern variations
StateImage submitButton = new StateImage.Builder()
    .addPatterns("submit-button.png", "submit-text.png", "ok-button.png")
    .setName("Submit Button")
    .build();

// Brobot automatically tries all patterns
ActionResult result = action.find(submitButton);
if (result.isSuccess()) {
    action.click(submitButton);
}

3. Use Descriptive Image Names​

Image names should be self-documenting since they appear in error logs:

  • ✅ login-submit-button.png
  • ✅ main-menu-file-option.png
  • ❌ img1.png
  • ❌ button.png

4. Monitor Logs in Production​

Set up log monitoring to catch image loading failures:

# application.properties
logging.level.io.github.jspinak.brobot.model.element.Pattern=ERROR

Configuration Options​

Logging Levels​

Control how image loading failures are reported:

# Show all image loading attempts
logging.level.io.github.jspinak.brobot.model.element.Pattern=DEBUG

# Show only failures (recommended for production)
logging.level.io.github.jspinak.brobot.model.element.Pattern=ERROR

# Suppress image loading logs entirely
logging.level.io.github.jspinak.brobot.model.element.Pattern=OFF

Mock Mode Configuration​

# Enable mock mode - no images are loaded
brobot.mock=true

# Disable mock mode - images are loaded from disk
brobot.mock=false

Testing Considerations​

Unit Tests​

  • Mock mode automatically skips image loading
  • Tests run without any image files present
  • Focus on logic, not image availability

Integration Tests​

  • Can verify image loading behavior
  • Test both success and failure paths
  • Ensure proper error logging occurs

Example Test​

import static org.junit.jupiter.api.Assertions.*;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.model.state.StateImage;
import io.github.jspinak.brobot.test.BrobotTestBase;

@SpringBootTest
public class FailSafeImageLoadingTest extends BrobotTestBase {

    @Autowired
    private Action action;

    @Test
    void shouldHandleMissingImageGracefully() {
        // Create StateImage with non-existent image
        StateImage stateImage = new StateImage.Builder()
            .addPatterns("non-existent.png")
            .setName("Non-existent Image")
            .build();

        // StateImage creation succeeds
        assertNotNull(stateImage);

        // StateImage contains patterns (even with null images internally)
        assertFalse(stateImage.getPatterns().isEmpty());

        // Find operations fail gracefully (no exceptions thrown)
        ActionResult result = action.find(stateImage);
        assertFalse(result.isSuccess());
    }
}

Summary​

The fail-safe image loading strategy ensures that Brobot automations are robust, maintainable, and production-ready. By handling missing images gracefully rather than failing fast, automations can:

  • Continue execution despite missing resources
  • Provide clear diagnostic information
  • Support gradual development and deployment
  • Maintain stability in production environments

This approach aligns with the reality of GUI automation where image availability cannot always be guaranteed, and partial functionality is often better than complete failure.

Testing & Mock Mode​

Pattern Management​

Error Handling & Debugging​

Configuration​

Core Concepts​

Capture System​