Image Find Debugging System
Comprehensive debugging system for troubleshooting image pattern matching issues in Brobot applications.
Overviewโ
The Image Find Debugging System provides detailed insights into why patterns may not be found during automation. It uses Spring AOP to intercept find operations and provides:
- Colorful console output with match details (Windows support via Jansi)
- Visual annotations on screenshots showing search regions and matches
- Comparison grids showing pattern vs found regions
- Session-based file organization for easy debugging
- Performance metrics and similarity scores
- HTML/JSON reports for analysis
Featuresโ
1. Colorful Console Outputโ
- Real-time feedback with ANSI colors
- Success/failure indicators with visual symbols (โ โ)
- Similarity scores and timing information
- Pattern dimensions and match counts
2. Visual Debuggingโ
- Annotated screenshots showing search regions
- Match highlights with similarity scores
- Comparison grids showing pattern vs matched regions
- Failed region indicators
3. Detailed Reportsโ
- HTML reports with interactive timeline
- JSON reports for programmatic analysis
- Session statistics and summaries
- Performance metrics
4. File Savingโ
- Screenshots of each find operation
- Pattern images for reference
- Visual comparison grids
- All outputs organized by session
Quick Startโ
Enable Debuggingโ
Add to your run command:
# Windows/Linux live automation with debugging
./gradlew bootRun --args='--spring.profiles.active=debug'
Debug Output Locationโ
Debug files are saved to:
debug/image-finding/
โโโ 2025-09-13_10-20-30/ # Session timestamp format: yyyy-MM-dd_HH-mm-ss
โ โโโ screenshots/
โ โ โโโ 001-LoginButton.png
โ โ โโโ 002-SubmitButton.png
โ โโโ patterns/
โ โ โโโ 001-LoginButton-pattern.png
โ โ โโโ 002-SubmitButton-pattern.png
โ โโโ comparisons/
โ โ โโโ 001-LoginButton-comparison.png
โ โ โโโ 002-SubmitButton-comparison.png
โ โโโ visual/
โ โ โโโ 001-LoginButton-annotated.png
โ โ โโโ 002-SubmitButton-annotated.png
โ โโโ logs/
โ โโโ session-summary.json
Configurationโ
Profile-Based Configurationโ
Brobot uses Spring profiles for clean configuration:
| Profile | Purpose | Properties File |
|---|---|---|
| (none) | Live automation | application.properties |
debug | Live with debugging | application-debug.properties |
mock | Testing without GUI | application-mock.properties |
mock,debug | Mock with debugging | Both profiles combined |
Key Propertiesโ
# Master switch
brobot.debug.image.enabled=true
# Debug level (OFF, BASIC, DETAILED, VISUAL, FULL)
brobot.debug.image.level=DETAILED
# File saving
brobot.debug.image.save-screenshots=true
brobot.debug.image.save-patterns=true
brobot.debug.image.save-comparisons=true
brobot.debug.image.output-dir=debug/image-finding
# Visual features
brobot.debug.image.visual.enabled=true
brobot.debug.image.visual.show-search-regions=true
brobot.debug.image.visual.show-match-scores=true
brobot.debug.image.visual.highlight-best-match=true
brobot.debug.image.visual.create-comparison-grid=true
# Console output (Windows support via Jansi)
brobot.debug.image.console.use-colors=true
brobot.debug.image.console.show-box=true
brobot.debug.image.console.show-timestamp=true
Note: Additional properties exist for advanced configuration including
visual.show-failed-regions,visual.create-heatmap, logging properties, and real-time monitoring. See the Properties Reference for a complete list.
Debug Levelsโ
OFFโ
No debugging output
BASICโ
- Success/failure status
- Basic timing information
DETAILEDโ
Everything from BASIC plus:
- Similarity scores
- Search parameters
- Match locations
- Failure reasons
VISUALโ
Everything from DETAILED plus:
- Visual annotations
- Screenshot saving
- Comparison grids
FULLโ
Everything from VISUAL plus:
- Memory usage
- Performance metrics
- Complete operation traces
Console Outputโ
Colorful Output on Windowsโ
The debug system uses the Jansi library to enable ANSI colors on Windows terminals:
โ
IMAGE FIND DEBUGGER: Session initialized
โ FIND START: LoginButton
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ FIND OPERATION DEBUG โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฃ
โ Pattern: LoginButton โ
โ Similarity: 0.85 โ
โ Status: โ SUCCESS โ
โ Best Match: 0.92 at (450, 320) โ
โ Time: 145ms โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Color Meaningsโ
- ๐ข Green: Successful matches
- ๐ด Red: Failed searches or errors
- ๐ก Yellow: Warnings or low similarity scores
- ๐ต Blue: Headers and important information
- โช Gray: Detailed/verbose information
Visual Outputโ
Annotated Screenshotsโ
- Blue dashed lines: Search regions
- Green rectangles: Successful matches
- Yellow rectangle: Best match
- Red X: Failed searches
Comparison Gridsโ
Side-by-side comparison of:
- Original pattern
- Found matches
- Similarity scores
Report Generationโ
HTML Reportsโ
Located at: debug/image-finding/{session-id}/reports/report.html
Contains:
- Session summary statistics
- Timeline of all operations
- Success rate metrics
- Interactive operation cards
JSON Reportsโ
Located at: debug/image-finding/{session-id}/reports/report.json
Structured data for:
- Automated analysis
- CI/CD integration
- Performance tracking
Troubleshooting Common Issuesโ
Images Not Foundโ
-
Check similarity threshold
Similarity is configured per-operation, not globally:
PatternFindOptions options = new PatternFindOptions.Builder()
.setSimilarity(0.7) // Lower for more lenient matching
.build();
ActionResult result = action.find(options, stateImage); -
Verify image format
- Use PNG format for best results
- Avoid JPEG for UI elements
- Ensure no scaling/compression
-
Review debug output
- Check "best score" in console
- Look at visual comparisons
- Examine failure reasons
Performance Issuesโ
-
Adjust debug level
brobot.debug.image.level=BASIC # Reduce overhead -
Disable file saving
brobot.debug.image.save-screenshots=false
brobot.debug.image.save-patterns=false
Issue: Images Cut with Windows Snipping Tool Not Foundโ
Symptoms: Patterns captured with Win+Shift+S aren't matching
Debug Steps:
- Enable DETAILED or VISUAL debugging:
brobot.debug.image.level=VISUAL
-
Check the debug output for:
- DPI differences between pattern and screen
- Color depth mismatches
- Scaling issues
-
Review the comparison grid to see visual differences
Common Solutions:
- Lower similarity threshold in your
PatternFindOptions.Builder().setSimilarity(0.7) - Check DPI settings in Windows Display Settings
- Ensure consistent color profiles
- Save patterns as PNG format
- Avoid resizing after capture
Issue: No Debug Output Appearingโ
Check:
- Correct profile is active:
# Look for: "The following 1 profile is active: "debug""
- Debug is enabled in properties:
brobot.debug.image.enabled=true
- AOP interceptor is initialized:
โ
FindOperationInterceptor initialized
Issue: No Colors in Console (Windows)โ
Solution: The Jansi library should auto-enable colors. If not:
- Check Jansi is in classpath:
implementation 'org.fusesource.jansi:jansi:2.4.0'
- Verify initialization message:
[Brobot Debug] Windows detected - ANSI colors enabled via Jansi
- Try different terminal (Windows Terminal, Git Bash, etc.)
Issue: ClassNotFoundException for ImageFindDebuggerโ
Symptoms: java.lang.ClassNotFoundException: io.github.jspinak.brobot.debug.ImageFindDebugger
Cause: Debug classes are conditional beans that only load when debugging is enabled.
Solution:
-
Enable debug mode in properties:
brobot.debug.image.enabled=true -
Check Spring Boot autoconfiguration:
- Ensure
@EnableAutoConfigurationor@SpringBootApplicationis present - Verify Brobot library is in classpath
- Ensure
-
Check for conditional bean issues:
// Debug classes use @ConditionalOnProperty
@ConditionalOnProperty(name = "brobot.debug.image.enabled", havingValue = "true") -
Verify in logs:
INFO Conditional bean ImageFindDebugger matched
INFO ImageFindDebugger initialized
Issue: AOP Not Intercepting Find Operationsโ
Symptoms: Debug output not appearing even when enabled, no interception logs
Cause: Spring AOP or AspectJ not properly configured
Solutions:
-
Ensure Spring AOP is enabled:
@SpringBootApplication
@EnableAspectJAutoProxy // Add this annotation
public class MyApplication {
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
}
} -
Verify AspectJ dependencies:
implementation 'org.springframework.boot:spring-boot-starter-aop'
implementation 'org.aspectj:aspectjweaver' -
Check AOP proxy creation:
# Enable CGLIB proxies if needed
spring.aop.proxy-target-class=true -
Verify interceptor initialization:
INFO FindOperationInterceptor initialized
INFO AOP pointcuts registered for Find.perform() -
Check method visibility:
- AOP requires public methods on Spring beans
- Direct
newinstantiation bypasses AOP proxies
Issue: Jansi Library Loading Errorsโ
Symptoms: java.lang.NoClassDefFoundError: org/fusesource/jansi/AnsiConsole
Cause: Jansi not in classpath or version conflict
Solutions:
-
Add Jansi dependency:
// In library/build.gradle
implementation 'org.fusesource.jansi:jansi:2.4.0' -
Check for version conflicts:
./gradlew dependencies --configuration runtimeClasspath | grep jansi -
Exclude conflicting versions:
configurations.all {
exclude group: 'org.fusesource.jansi', module: 'jansi'
}
dependencies {
implementation 'org.fusesource.jansi:jansi:2.4.0'
} -
Fallback: Disable colors:
brobot.debug.image.console.use-colors=false
Issue: Debug Files Not Savingโ
Symptoms: Console output works but no files in debug/image-finding/ directory
Cause: File permissions, path issues, or file saving disabled
Solutions:
-
Verify file saving is enabled:
brobot.debug.image.save-screenshots=true
brobot.debug.image.save-patterns=true
brobot.debug.image.save-comparisons=true -
Check output directory path:
# Use absolute path if relative path fails
brobot.debug.image.output-dir=/absolute/path/to/debug/image-finding -
Verify write permissions:
# Check directory exists and is writable
ls -la debug/
mkdir -p debug/image-finding
chmod 755 debug/image-finding -
Check disk space:
- Debug files can be large (screenshots, comparisons)
- Ensure sufficient disk space available
-
Review logs for I/O errors:
ERROR Failed to save screenshot: /path/to/file.png
java.io.IOException: Permission denied
Integration with CI/CDโ
For comprehensive CI/CD testing strategies, see the CI/CD Testing Guide.
GitHub Actionsโ
- name: Run tests with debug
run: ./gradlew test --args='--spring.profiles.active=debug'
- name: Upload debug artifacts
if: failure()
uses: actions/upload-artifact@v2
with:
name: debug-reports
path: debug/image-finding/**
Jenkinsโ
stage('Test with Debug') {
steps {
sh './gradlew test -Dspring.profiles.active=debug'
}
post {
failure {
archiveArtifacts artifacts: 'debug/image-finding/**'
}
}
}
API Usageโ
Programmatic Accessโ
Complete example showing session management. For more information on StateImage and state components, see the States Guide.
package com.example.brobot.debugging;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import io.github.jspinak.brobot.debug.ImageFindDebugger;
import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.model.state.StateImage;
@Component
public class ImageDebugExample {
@Autowired
private ImageFindDebugger debugger;
@Autowired
private Action action;
public void debugImageFind() {
// Example pattern to find
StateImage stateImage = new StateImage.Builder()
.addPatterns("button-pattern")
.build();
try {
// Initialize session
debugger.initializeSession();
// Debug operations are automatically intercepted
ActionResult result = action.find(stateImage);
} finally {
// Always finalize session, even if exception occurs
debugger.finalizeSession();
}
}
}
Custom Debug Infoโ
Access detailed debugging information programmatically:
package com.example.brobot.debugging;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import io.github.jspinak.brobot.debug.ImageFindDebugger;
import io.github.jspinak.brobot.debug.ImageFindDebugger.FindDebugInfo;
import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.action.ObjectCollection;
import io.github.jspinak.brobot.action.basic.find.PatternFindOptions;
import io.github.jspinak.brobot.model.state.StateImage;
@Component
public class DebugInfoExample {
@Autowired
private ImageFindDebugger debugger;
@Autowired
private Action action;
public void getDebugInfo() {
// Create pattern to find
StateImage stateImage = new StateImage.Builder()
.addPatterns("login-button")
.build();
// Create object collection
ObjectCollection objectCollection = new ObjectCollection.Builder()
.withImages(stateImage)
.build();
// Configure find options
PatternFindOptions options = new PatternFindOptions.Builder()
.setSimilarity(0.85)
.build();
// Perform find operation
ActionResult result = action.find(options, objectCollection);
// Get detailed debug info
FindDebugInfo debugInfo =
debugger.debugFindOperation(objectCollection, options, result);
// Access debug information (getters are auto-generated by Lombok)
System.out.println("Operation ID: " + debugInfo.getOperationId());
System.out.println("Best Score: " + debugInfo.getBestScore());
System.out.println("Duration: " + debugInfo.getSearchDuration() + "ms");
System.out.println("Found: " + debugInfo.isFound());
System.out.println("Match Count: " + debugInfo.getMatchCount());
}
}
Best Practicesโ
- Development: Use DETAILED or VISUAL level
- Testing: Use BASIC level with file saving disabled
- Production: Keep debugging OFF or use BASIC for critical operations
- CI/CD: Enable on failure with artifact collection
Performance Impactโ
| Level | Console Output | File I/O | Performance Impact |
|---|---|---|---|
| OFF | None | None | 0% |
| BASIC | Minimal | None | ~2% |
| DETAILED | Moderate | None | ~5% |
| VISUAL | Heavy | Heavy | ~15-20% |
| FULL | Very Heavy | Very Heavy | ~25-30% |
Note: Performance impact percentages are approximate and vary based on system configuration, image sizes, screen resolution, pattern complexity, and operation types. Use these as general guidelines, not exact measurements. Actual impact may be higher on slower systems or when processing large images.
Architectureโ
Componentsโ
- ImageDebugConfig: Configuration management via Spring properties
- FindOperationInterceptor: AOP aspect intercepting find operations
- ImageFindDebugger: Core orchestrator for debug operations
- VisualDebugRenderer: Creates annotated images and comparisons
- AnsiColor: Console coloring with Windows support via Jansi
Spring AOP Integrationโ
The system uses Spring AOP (see AspectJ Usage Guide for advanced AOP patterns) with @Aspect and @Around advice to intercept:
Find.perform()operationsFindPipeline.saveMatchesToStateImages()calls
This provides transparent debugging without code changes.
Advanced Featuresโ
Programmatic Controlโ
Dynamically control debugging at runtime:
package com.example.brobot.debugging;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import io.github.jspinak.brobot.debug.ImageDebugConfig;
import io.github.jspinak.brobot.debug.ImageDebugConfig.DebugLevel;
import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.model.state.StateImage;
@Component
public class ProgrammaticDebugControl {
@Autowired
private ImageDebugConfig debugConfig;
@Autowired
private Action action;
public void temporaryDebugMode() {
// Save original settings
boolean originalEnabled = debugConfig.isEnabled();
DebugLevel originalLevel = debugConfig.getLevel();
try {
// Temporarily enable debugging
debugConfig.setEnabled(true);
debugConfig.setLevel(DebugLevel.VISUAL);
// Example pattern to find
StateImage stateImage = new StateImage.Builder()
.addPatterns("login-button")
.build();
// Perform find operation with debugging enabled
ActionResult result = action.find(stateImage);
} finally {
// Always restore original settings
debugConfig.setEnabled(originalEnabled);
debugConfig.setLevel(originalLevel);
}
}
}
Related Documentationโ
- Properties Reference - Complete property list
- DPI Resolution Guide - Understanding DPI issues
- Pattern Matching Guide - Pattern matching troubleshooting
- Mock Mode Guide - Testing without GUI