Skip to main content
Version: Latest

Image Find Debugging System

Comprehensive debugging system for troubleshooting image pattern matching issues in Brobot applications.

Overviewโ€‹

The Image Find Debugging System provides detailed insights into why patterns may not be found during automation. It uses Spring AOP to intercept find operations and provides:

  • Colorful console output with match details (Windows support via Jansi)
  • Visual annotations on screenshots showing search regions and matches
  • Comparison grids showing pattern vs found regions
  • Session-based file organization for easy debugging
  • Performance metrics and similarity scores
  • HTML/JSON reports for analysis

Featuresโ€‹

1. Colorful Console Outputโ€‹

  • Real-time feedback with ANSI colors
  • Success/failure indicators with visual symbols (โœ… โŒ)
  • Similarity scores and timing information
  • Pattern dimensions and match counts

2. Visual Debuggingโ€‹

  • Annotated screenshots showing search regions
  • Match highlights with similarity scores
  • Comparison grids showing pattern vs matched regions
  • Failed region indicators

3. Detailed Reportsโ€‹

  • HTML reports with interactive timeline
  • JSON reports for programmatic analysis
  • Session statistics and summaries
  • Performance metrics

4. File Savingโ€‹

  • Screenshots of each find operation
  • Pattern images for reference
  • Visual comparison grids
  • All outputs organized by session

Quick Startโ€‹

Enable Debuggingโ€‹

Add to your run command:

# Windows/Linux live automation with debugging
./gradlew bootRun --args='--spring.profiles.active=debug'

Debug Output Locationโ€‹

Debug files are saved to:

debug/image-finding/
โ”œโ”€โ”€ 2025-09-13_10-20-30/ # Session timestamp format: yyyy-MM-dd_HH-mm-ss
โ”‚ โ”œโ”€โ”€ screenshots/
โ”‚ โ”‚ โ”œโ”€โ”€ 001-LoginButton.png
โ”‚ โ”‚ โ””โ”€โ”€ 002-SubmitButton.png
โ”‚ โ”œโ”€โ”€ patterns/
โ”‚ โ”‚ โ”œโ”€โ”€ 001-LoginButton-pattern.png
โ”‚ โ”‚ โ””โ”€โ”€ 002-SubmitButton-pattern.png
โ”‚ โ”œโ”€โ”€ comparisons/
โ”‚ โ”‚ โ”œโ”€โ”€ 001-LoginButton-comparison.png
โ”‚ โ”‚ โ””โ”€โ”€ 002-SubmitButton-comparison.png
โ”‚ โ”œโ”€โ”€ visual/
โ”‚ โ”‚ โ”œโ”€โ”€ 001-LoginButton-annotated.png
โ”‚ โ”‚ โ””โ”€โ”€ 002-SubmitButton-annotated.png
โ”‚ โ””โ”€โ”€ logs/
โ”‚ โ””โ”€โ”€ session-summary.json

Configurationโ€‹

Profile-Based Configurationโ€‹

Brobot uses Spring profiles for clean configuration:

ProfilePurposeProperties File
(none)Live automationapplication.properties
debugLive with debuggingapplication-debug.properties
mockTesting without GUIapplication-mock.properties
mock,debugMock with debuggingBoth profiles combined

Key Propertiesโ€‹

# Master switch
brobot.debug.image.enabled=true

# Debug level (OFF, BASIC, DETAILED, VISUAL, FULL)
brobot.debug.image.level=DETAILED

# File saving
brobot.debug.image.save-screenshots=true
brobot.debug.image.save-patterns=true
brobot.debug.image.save-comparisons=true
brobot.debug.image.output-dir=debug/image-finding

# Visual features
brobot.debug.image.visual.enabled=true
brobot.debug.image.visual.show-search-regions=true
brobot.debug.image.visual.show-match-scores=true
brobot.debug.image.visual.highlight-best-match=true
brobot.debug.image.visual.create-comparison-grid=true

# Console output (Windows support via Jansi)
brobot.debug.image.console.use-colors=true
brobot.debug.image.console.show-box=true
brobot.debug.image.console.show-timestamp=true

Note: Additional properties exist for advanced configuration including visual.show-failed-regions, visual.create-heatmap, logging properties, and real-time monitoring. See the Properties Reference for a complete list.

Debug Levelsโ€‹

OFFโ€‹

No debugging output

BASICโ€‹

  • Success/failure status
  • Basic timing information

DETAILEDโ€‹

Everything from BASIC plus:

  • Similarity scores
  • Search parameters
  • Match locations
  • Failure reasons

VISUALโ€‹

Everything from DETAILED plus:

  • Visual annotations
  • Screenshot saving
  • Comparison grids

FULLโ€‹

Everything from VISUAL plus:

  • Memory usage
  • Performance metrics
  • Complete operation traces

Console Outputโ€‹

Colorful Output on Windowsโ€‹

The debug system uses the Jansi library to enable ANSI colors on Windows terminals:

โœ… IMAGE FIND DEBUGGER: Session initialized
โ†’ FIND START: LoginButton
โ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•—
โ•‘ FIND OPERATION DEBUG โ•‘
โ• โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•ฃ
โ•‘ Pattern: LoginButton โ•‘
โ•‘ Similarity: 0.85 โ•‘
โ•‘ Status: โœ“ SUCCESS โ•‘
โ•‘ Best Match: 0.92 at (450, 320) โ•‘
โ•‘ Time: 145ms โ•‘
โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•

Color Meaningsโ€‹

  • ๐ŸŸข Green: Successful matches
  • ๐Ÿ”ด Red: Failed searches or errors
  • ๐ŸŸก Yellow: Warnings or low similarity scores
  • ๐Ÿ”ต Blue: Headers and important information
  • โšช Gray: Detailed/verbose information

Visual Outputโ€‹

Annotated Screenshotsโ€‹

  • Blue dashed lines: Search regions
  • Green rectangles: Successful matches
  • Yellow rectangle: Best match
  • Red X: Failed searches

Comparison Gridsโ€‹

Side-by-side comparison of:

  • Original pattern
  • Found matches
  • Similarity scores

Report Generationโ€‹

HTML Reportsโ€‹

Located at: debug/image-finding/{session-id}/reports/report.html

Contains:

  • Session summary statistics
  • Timeline of all operations
  • Success rate metrics
  • Interactive operation cards

JSON Reportsโ€‹

Located at: debug/image-finding/{session-id}/reports/report.json

Structured data for:

  • Automated analysis
  • CI/CD integration
  • Performance tracking

Troubleshooting Common Issuesโ€‹

Images Not Foundโ€‹

  1. Check similarity threshold

    Similarity is configured per-operation, not globally:

    PatternFindOptions options = new PatternFindOptions.Builder()
    .setSimilarity(0.7) // Lower for more lenient matching
    .build();

    ActionResult result = action.find(options, stateImage);
  2. Verify image format

    • Use PNG format for best results
    • Avoid JPEG for UI elements
    • Ensure no scaling/compression
  3. Review debug output

    • Check "best score" in console
    • Look at visual comparisons
    • Examine failure reasons

Performance Issuesโ€‹

  1. Adjust debug level

    brobot.debug.image.level=BASIC  # Reduce overhead
  2. Disable file saving

    brobot.debug.image.save-screenshots=false
    brobot.debug.image.save-patterns=false

Issue: Images Cut with Windows Snipping Tool Not Foundโ€‹

Symptoms: Patterns captured with Win+Shift+S aren't matching

Debug Steps:

  1. Enable DETAILED or VISUAL debugging:
brobot.debug.image.level=VISUAL
  1. Check the debug output for:

    • DPI differences between pattern and screen
    • Color depth mismatches
    • Scaling issues
  2. Review the comparison grid to see visual differences

Common Solutions:

  • Lower similarity threshold in your PatternFindOptions.Builder().setSimilarity(0.7)
  • Check DPI settings in Windows Display Settings
  • Ensure consistent color profiles
  • Save patterns as PNG format
  • Avoid resizing after capture

Issue: No Debug Output Appearingโ€‹

Check:

  1. Correct profile is active:
# Look for: "The following 1 profile is active: "debug""
  1. Debug is enabled in properties:
brobot.debug.image.enabled=true
  1. AOP interceptor is initialized:
โœ… FindOperationInterceptor initialized

Issue: No Colors in Console (Windows)โ€‹

Solution: The Jansi library should auto-enable colors. If not:

  1. Check Jansi is in classpath:
implementation 'org.fusesource.jansi:jansi:2.4.0'
  1. Verify initialization message:
[Brobot Debug] Windows detected - ANSI colors enabled via Jansi
  1. Try different terminal (Windows Terminal, Git Bash, etc.)

Issue: ClassNotFoundException for ImageFindDebuggerโ€‹

Symptoms: java.lang.ClassNotFoundException: io.github.jspinak.brobot.debug.ImageFindDebugger

Cause: Debug classes are conditional beans that only load when debugging is enabled.

Solution:

  1. Enable debug mode in properties:

    brobot.debug.image.enabled=true
  2. Check Spring Boot autoconfiguration:

    • Ensure @EnableAutoConfiguration or @SpringBootApplication is present
    • Verify Brobot library is in classpath
  3. Check for conditional bean issues:

    // Debug classes use @ConditionalOnProperty
    @ConditionalOnProperty(name = "brobot.debug.image.enabled", havingValue = "true")
  4. Verify in logs:

    INFO  Conditional bean ImageFindDebugger matched
    INFO ImageFindDebugger initialized

Issue: AOP Not Intercepting Find Operationsโ€‹

Symptoms: Debug output not appearing even when enabled, no interception logs

Cause: Spring AOP or AspectJ not properly configured

Solutions:

  1. Ensure Spring AOP is enabled:

    @SpringBootApplication
    @EnableAspectJAutoProxy // Add this annotation
    public class MyApplication {
    public static void main(String[] args) {
    SpringApplication.run(MyApplication.class, args);
    }
    }
  2. Verify AspectJ dependencies:

    implementation 'org.springframework.boot:spring-boot-starter-aop'
    implementation 'org.aspectj:aspectjweaver'
  3. Check AOP proxy creation:

    # Enable CGLIB proxies if needed
    spring.aop.proxy-target-class=true
  4. Verify interceptor initialization:

    INFO  FindOperationInterceptor initialized
    INFO AOP pointcuts registered for Find.perform()
  5. Check method visibility:

    • AOP requires public methods on Spring beans
    • Direct new instantiation bypasses AOP proxies

Issue: Jansi Library Loading Errorsโ€‹

Symptoms: java.lang.NoClassDefFoundError: org/fusesource/jansi/AnsiConsole

Cause: Jansi not in classpath or version conflict

Solutions:

  1. Add Jansi dependency:

    // In library/build.gradle
    implementation 'org.fusesource.jansi:jansi:2.4.0'
  2. Check for version conflicts:

    ./gradlew dependencies --configuration runtimeClasspath | grep jansi
  3. Exclude conflicting versions:

    configurations.all {
    exclude group: 'org.fusesource.jansi', module: 'jansi'
    }
    dependencies {
    implementation 'org.fusesource.jansi:jansi:2.4.0'
    }
  4. Fallback: Disable colors:

    brobot.debug.image.console.use-colors=false

Issue: Debug Files Not Savingโ€‹

Symptoms: Console output works but no files in debug/image-finding/ directory

Cause: File permissions, path issues, or file saving disabled

Solutions:

  1. Verify file saving is enabled:

    brobot.debug.image.save-screenshots=true
    brobot.debug.image.save-patterns=true
    brobot.debug.image.save-comparisons=true
  2. Check output directory path:

    # Use absolute path if relative path fails
    brobot.debug.image.output-dir=/absolute/path/to/debug/image-finding
  3. Verify write permissions:

    # Check directory exists and is writable
    ls -la debug/
    mkdir -p debug/image-finding
    chmod 755 debug/image-finding
  4. Check disk space:

    • Debug files can be large (screenshots, comparisons)
    • Ensure sufficient disk space available
  5. Review logs for I/O errors:

    ERROR Failed to save screenshot: /path/to/file.png
    java.io.IOException: Permission denied

Integration with CI/CDโ€‹

For comprehensive CI/CD testing strategies, see the CI/CD Testing Guide.

GitHub Actionsโ€‹

- name: Run tests with debug
run: ./gradlew test --args='--spring.profiles.active=debug'

- name: Upload debug artifacts
if: failure()
uses: actions/upload-artifact@v2
with:
name: debug-reports
path: debug/image-finding/**

Jenkinsโ€‹

stage('Test with Debug') {
steps {
sh './gradlew test -Dspring.profiles.active=debug'
}
post {
failure {
archiveArtifacts artifacts: 'debug/image-finding/**'
}
}
}

API Usageโ€‹

Programmatic Accessโ€‹

Complete example showing session management. For more information on StateImage and state components, see the States Guide.

package com.example.brobot.debugging;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import io.github.jspinak.brobot.debug.ImageFindDebugger;
import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.model.state.StateImage;

@Component
public class ImageDebugExample {

@Autowired
private ImageFindDebugger debugger;

@Autowired
private Action action;

public void debugImageFind() {
// Example pattern to find
StateImage stateImage = new StateImage.Builder()
.addPatterns("button-pattern")
.build();

try {
// Initialize session
debugger.initializeSession();

// Debug operations are automatically intercepted
ActionResult result = action.find(stateImage);

} finally {
// Always finalize session, even if exception occurs
debugger.finalizeSession();
}
}
}

Custom Debug Infoโ€‹

Access detailed debugging information programmatically:

package com.example.brobot.debugging;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import io.github.jspinak.brobot.debug.ImageFindDebugger;
import io.github.jspinak.brobot.debug.ImageFindDebugger.FindDebugInfo;
import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.action.ObjectCollection;
import io.github.jspinak.brobot.action.basic.find.PatternFindOptions;
import io.github.jspinak.brobot.model.state.StateImage;

@Component
public class DebugInfoExample {

@Autowired
private ImageFindDebugger debugger;

@Autowired
private Action action;

public void getDebugInfo() {
// Create pattern to find
StateImage stateImage = new StateImage.Builder()
.addPatterns("login-button")
.build();

// Create object collection
ObjectCollection objectCollection = new ObjectCollection.Builder()
.withImages(stateImage)
.build();

// Configure find options
PatternFindOptions options = new PatternFindOptions.Builder()
.setSimilarity(0.85)
.build();

// Perform find operation
ActionResult result = action.find(options, objectCollection);

// Get detailed debug info
FindDebugInfo debugInfo =
debugger.debugFindOperation(objectCollection, options, result);

// Access debug information (getters are auto-generated by Lombok)
System.out.println("Operation ID: " + debugInfo.getOperationId());
System.out.println("Best Score: " + debugInfo.getBestScore());
System.out.println("Duration: " + debugInfo.getSearchDuration() + "ms");
System.out.println("Found: " + debugInfo.isFound());
System.out.println("Match Count: " + debugInfo.getMatchCount());
}
}

Best Practicesโ€‹

  1. Development: Use DETAILED or VISUAL level
  2. Testing: Use BASIC level with file saving disabled
  3. Production: Keep debugging OFF or use BASIC for critical operations
  4. CI/CD: Enable on failure with artifact collection

Performance Impactโ€‹

LevelConsole OutputFile I/OPerformance Impact
OFFNoneNone0%
BASICMinimalNone~2%
DETAILEDModerateNone~5%
VISUALHeavyHeavy~15-20%
FULLVery HeavyVery Heavy~25-30%

Note: Performance impact percentages are approximate and vary based on system configuration, image sizes, screen resolution, pattern complexity, and operation types. Use these as general guidelines, not exact measurements. Actual impact may be higher on slower systems or when processing large images.

Architectureโ€‹

Componentsโ€‹

  1. ImageDebugConfig: Configuration management via Spring properties
  2. FindOperationInterceptor: AOP aspect intercepting find operations
  3. ImageFindDebugger: Core orchestrator for debug operations
  4. VisualDebugRenderer: Creates annotated images and comparisons
  5. AnsiColor: Console coloring with Windows support via Jansi

Spring AOP Integrationโ€‹

The system uses Spring AOP (see AspectJ Usage Guide for advanced AOP patterns) with @Aspect and @Around advice to intercept:

  • Find.perform() operations
  • FindPipeline.saveMatchesToStateImages() calls

This provides transparent debugging without code changes.

Advanced Featuresโ€‹

Programmatic Controlโ€‹

Dynamically control debugging at runtime:

package com.example.brobot.debugging;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import io.github.jspinak.brobot.debug.ImageDebugConfig;
import io.github.jspinak.brobot.debug.ImageDebugConfig.DebugLevel;
import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.model.state.StateImage;

@Component
public class ProgrammaticDebugControl {

@Autowired
private ImageDebugConfig debugConfig;

@Autowired
private Action action;

public void temporaryDebugMode() {
// Save original settings
boolean originalEnabled = debugConfig.isEnabled();
DebugLevel originalLevel = debugConfig.getLevel();

try {
// Temporarily enable debugging
debugConfig.setEnabled(true);
debugConfig.setLevel(DebugLevel.VISUAL);

// Example pattern to find
StateImage stateImage = new StateImage.Builder()
.addPatterns("login-button")
.build();

// Perform find operation with debugging enabled
ActionResult result = action.find(stateImage);

} finally {
// Always restore original settings
debugConfig.setEnabled(originalEnabled);
debugConfig.setLevel(originalLevel);
}
}
}