Version: Latest

Advanced Illustration System

Brobot's advanced illustration system provides intelligent, context-aware visual documentation of automation actions with performance optimization and quality-based filtering.

Overview

The enhanced illustration system includes:

Context-aware illustration decisions based on action history and system state
Performance optimization with adaptive sampling and batching
Quality-based filtering to focus on meaningful visualizations
Granular configuration for different environments and use cases
Resource management to prevent system overload

Configuration System

Basic Configuration

@Configuration
public class IllustrationConfig {
    
    @Bean
    public IllustrationConfig illustrationConfig() {
        return IllustrationConfig.builder()
            .globalEnabled(true)
            .actionEnabled(ActionOptions.Action.FIND, true)
            .actionEnabled(ActionOptions.Action.CLICK, true)
            .actionEnabled(ActionOptions.Action.MOVE, false) // Disable noisy move illustrations
            .qualityThreshold(0.75) // Only illustrate high-quality matches
            .maxIllustrationsPerMinute(30) // Rate limiting
            .build();
    }
}

Context-Based Filtering

@Test
public void configureContextFilters() {
    IllustrationConfig config = IllustrationConfig.builder()
        .globalEnabled(true)
        // Only illustrate failures for debugging
        .contextFilter("failures_only", context -> 
            context.getLastActionResult() != null && 
            !context.getLastActionResult().isSuccess())
        // Only illustrate first occurrence of each action type
        .contextFilter("first_occurrence", context -> 
            context.isFirstExecution())
        // Only illustrate during retry attempts
        .contextFilter("retries_only", context -> 
            context.isRetryAttempt())
        .build();
        
    illustrationController.setConfig(config);
}

Adaptive Sampling

@Configuration
public class PerformanceOptimizedIllustrations {
    
    @Bean
    public IllustrationConfig performanceConfig() {
        return IllustrationConfig.builder()
            .globalEnabled(true)
            .adaptiveSampling(true) // Enable intelligent sampling
            // High-frequency actions get reduced sampling
            .samplingRate(ActionOptions.Action.FIND, 1.0)   // Always illustrate find
            .samplingRate(ActionOptions.Action.MOVE, 0.1)   // Sample 10% of moves
            .samplingRate(ActionOptions.Action.CLICK, 0.5)  // Sample 50% of clicks
            // Performance-based batching
            .batchConfig(BatchConfig.builder()
                .maxBatchSize(20)
                .flushInterval(Duration.ofSeconds(10))
                .flushOnStateTransition(true)
                .maxMemoryUsageMB(50)
                .build())
            .build();
    }
}

Quality-Based Filtering

Basic Quality Metrics

IllustrationConfig qualityFocusedConfig = IllustrationConfig.builder()
    .qualityThreshold(0.8) // Only illustrate matches with >80% quality
    .qualityMetrics(QualityMetrics.builder()
        .minSimilarity(0.75)      // Minimum image similarity
        .minConfidence(0.6)       // Minimum match confidence
        .useRegionSize(true)      // Consider region size in quality
        .useExecutionTime(false)  // Don't factor in timing
        .build())
    .build();

Custom Quality Calculation

IllustrationConfig customQualityConfig = IllustrationConfig.builder()
    .qualityThreshold(0.7)
    .qualityMetrics(QualityMetrics.builder()
        .customQualityCalculator(context -> {
            ActionResult result = context.getLastActionResult();
            if (result == null || result.getMatchList().isEmpty()) {
                return 0.0;
            }
            
            // Custom quality calculation
            double avgSimilarity = result.getMatchList().stream()
                .mapToDouble(match -> match.getScore())
                .average().orElse(0.0);
                
            // Boost quality for first executions
            if (context.isFirstExecution()) {
                avgSimilarity *= 1.2;
            }
            
            // Reduce quality for frequent actions
            if (context.getRecentIllustrationCount() > 10) {
                avgSimilarity *= 0.8;
            }
            
            return Math.min(1.0, avgSimilarity);
        })
        .build())
    .build();

Performance Optimization

Smart Sampling Strategies

@Service
public class CustomIllustrationStrategy {
    
    @Autowired
    private IllustrationPerformanceOptimizer optimizer;
    
    public void configureAdaptiveSampling() {
        IllustrationConfig config = IllustrationConfig.builder()
            .adaptiveSampling(true)
            .samplingRate(ActionOptions.Action.FIND, 1.0)
            .contextFilter("performance_aware", context -> {
                // Skip during high system load
                if (context.getSystemMetrics() != null && 
                    context.getSystemMetrics().isHighLoad()) {
                    return context.getPriority() == IllustrationContext.Priority.CRITICAL;
                }
                
                // Reduce frequency for repeated actions
                if (context.getConsecutiveFailures() > 0) {
                    return true; // Always illustrate during failures
                }
                
                // Sample based on recent success rate
                return context.getRecentSuccessRate() < 0.9; // Illustrate when success rate drops
            })
            .build();
            
        optimizer.setConfig(config);
    }
}

Batching for Performance

@Test 
public void testBatchedIllustrations() {
    IllustrationConfig config = IllustrationConfig.builder()
        .batchConfig(BatchConfig.builder()
            .maxBatchSize(50)                    // Batch up to 50 illustrations
            .flushInterval(Duration.ofSeconds(30)) // Flush every 30 seconds
            .flushOnStateTransition(true)        // Flush when states change
            .maxMemoryUsageMB(100)               // Memory limit for batching
            .build())
        .contextFilter("batch_eligible", context -> {
            // Only batch low-priority, frequent actions
            return context.getPriority() == IllustrationContext.Priority.LOW &&
                   context.getCurrentAction() == ActionOptions.Action.MOVE;
        })
        .build();
        
    illustrationController.setConfig(config);
    
    // Execute many actions - some will be batched for efficiency
    for (int i = 0; i < 100; i++) {
        actions.move(randomLocation());
    }
    
    // Verify performance optimization worked
    PerformanceMetrics.MetricsSnapshot metrics = 
        illustrationPerformanceOptimizer.getPerformanceMetrics();
    
    assertTrue("Some illustrations should be batched", 
               metrics.getIllustrationsBatched() > 0);
    assertTrue("Skip rate should be reasonable", 
               metrics.getSkipRate() < 0.8);
}

State-Aware Illustrations

Priority-Based Illustration

IllustrationConfig stateAwareConfig = IllustrationConfig.builder()
    // Always illustrate critical states
    .alwaysIllustrateState("ERROR_STATE")
    .alwaysIllustrateState("LOGIN_FAILURE")
    .alwaysIllustrateState("PAYMENT_CONFIRMATION")
    
    // Never illustrate noisy intermediate states
    .neverIllustrateAction(ActionOptions.Action.MOVE)
    
    .contextFilter("state_priority", context -> {
        // High priority for error conditions
        if (context.hasActiveState("ERROR_STATE", "WARNING_STATE")) {
            return true;
        }
        
        // Medium priority for authentication flows
        if (context.hasActiveState("LOGIN_STATE", "AUTHENTICATION")) {
            return context.getConsecutiveFailures() > 0; // Only on failures
        }
        
        // Low priority for routine operations
        return context.getRecentSuccessRate() < 0.95;
    })
    .build();

Integration with Existing Code

Migrating from Basic Illustrations

// Old approach - simple enable/disable
@Configuration
public class OldIllustrationConfig {
    @Bean
    public BrobotSettings settings() {
        BrobotSettings settings = new BrobotSettings();
        settings.drawFind = true;
        settings.drawClick = true; 
        settings.saveHistory = true;
        return settings;
    }
}

// New approach - context-aware configuration
@Configuration
public class NewIllustrationConfig {
    @Bean
    public IllustrationConfig illustrationConfig() {
        return IllustrationConfig.builder()
            .globalEnabled(true)
            .actionEnabled(ActionOptions.Action.FIND, true)
            .actionEnabled(ActionOptions.Action.CLICK, true)
            
            // Add intelligent filtering
            .contextFilter("meaningful_only", context -> 
                context.isFirstExecution() || 
                context.getConsecutiveFailures() > 0 ||
                context.getLastActionResult() != null && 
                !context.getLastActionResult().isSuccess())
                
            // Add performance optimization    
            .samplingRate(ActionOptions.Action.FIND, 0.8)
            .qualityThreshold(0.7)
            .maxIllustrationsPerMinute(60)
            .build();
    }
}

ActionConfig Integration

The new system seamlessly works with both ActionOptions and ActionConfig:

// Works with traditional ActionOptions
ActionOptions options = new ActionOptions.Builder()
    .setAction(ActionOptions.Action.FIND)
    .setIllustrate(ActionOptions.Illustrate.YES)
    .build();

// Also works with new ActionConfig
PatternFindOptions findConfig = new PatternFindOptions.Builder()
    .setIllustrate(ActionConfig.Illustrate.USE_GLOBAL)
    .setSimilarity(0.8)
    .build();
    
// Both will respect the advanced configuration rules
ActionResult result1 = actions.find(element).configure(options);
ActionResult result2 = actions.find(element).configure(findConfig);

Monitoring and Debugging

Performance Metrics

@Test
public void monitorIllustrationPerformance() {
    // Execute some actions
    for (int i = 0; i < 100; i++) {
        actions.find(testElement);
    }
    
    // Get performance metrics
    PerformanceMetrics.MetricsSnapshot metrics = 
        illustrationPerformanceOptimizer.getPerformanceMetrics();
        
    System.out.println("Illustrations per minute: " + metrics.getIllustrationsPerMinute());
    System.out.println("Skip rate: " + (metrics.getSkipRate() * 100) + "%");
    System.out.println("High quality rate: " + (metrics.getHighQualityRate() * 100) + "%");
    System.out.println("Average processing time: " + metrics.getAverageProcessingTimeMs() + "ms");
    System.out.println("Average memory usage: " + metrics.getAverageMemoryUsageMB() + "MB");
    
    // Verify performance is within acceptable bounds
    assertTrue("Processing time should be reasonable", 
               metrics.getAverageProcessingTimeMs() < 500);
    assertTrue("Memory usage should be controlled", 
               metrics.getAverageMemoryUsageMB() < 10);
}

Debugging Configuration

IllustrationConfig debugConfig = IllustrationConfig.builder()
    .globalEnabled(true)
    // Debug mode - illustrate everything with detailed context
    .contextFilter("debug_mode", context -> {
        System.out.printf("Action: %s, Success: %s, Priority: %s%n", 
            context.getCurrentAction(),
            context.getLastActionResult() != null ? 
                context.getLastActionResult().isSuccess() : "N/A",
            context.getPriority());
        return true; // Illustrate everything in debug mode
    })
    .qualityThreshold(0.0) // No quality filtering in debug
    .maxIllustrationsPerMinute(Integer.MAX_VALUE) // No rate limiting
    .build();

Best Practices

Configuration Strategy

Start with defaults - Begin with sensible defaults and tune based on needs
Monitor performance - Use metrics to identify bottlenecks
Test different environments - Configurations may need adjustment for different systems
Use state awareness - Leverage state information for smarter illustration decisions

Performance Guidelines

Use sampling for high-frequency actions - Reduce overhead for repetitive operations
Enable batching for bulk operations - Improve I/O efficiency
Set appropriate quality thresholds - Focus on meaningful illustrations
Monitor system resources - Prevent illustration system from impacting automation performance

Quality Management

Define clear quality metrics - Establish what constitutes a valuable illustration
Use custom quality calculators - Tailor quality assessment to your specific needs
Balance coverage and noise - Ensure important events are captured without overwhelming detail
Regular quality audits - Periodically review illustration quality and adjust thresholds

Context Utilization

Leverage action history - Use past executions to inform current decisions
Consider system state - Factor in current states and transitions
Use priority levels - Ensure critical actions are always illustrated
Implement fallbacks - Have sensible defaults when context is unavailable

Configuration Reference

IllustrationConfig Properties

Property	Type	Default	Description
`globalEnabled`	boolean	true	Master switch for all illustrations
`actionEnabledMap`	`Map<Action, Boolean>`	all true	Per-action enablement
`contextFilters`	`Map<String, Predicate>`	empty	Context-based filters
`samplingRates`	`Map<Action, Double>`	all 1.0	Action sampling rates
`qualityThreshold`	double	0.0	Minimum quality for illustration
`maxIllustrationsPerMinute`	int	MAX_VALUE	Rate limiting
`adaptiveSampling`	boolean	false	Enable adaptive sampling

BatchConfig Properties

Property	Type	Default	Description
`maxBatchSize`	int	10	Maximum illustrations per batch
`flushInterval`	Duration	5s	Maximum batch hold time
`flushOnStateTransition`	boolean	true	Flush on state changes
`maxMemoryUsageMB`	int	100	Memory limit for batching

QualityMetrics Properties

Property	Type	Default	Description
`minSimilarity`	double	0.7	Minimum match similarity
`minConfidence`	double	0.5	Minimum match confidence
`useRegionSize`	boolean	true	Factor in region size
`useExecutionTime`	boolean	false	Factor in execution time
`customQualityCalculator`	Function	null	Custom quality function

The advanced illustration system provides powerful tools for creating meaningful, performance-optimized visual documentation of your automation execution while maintaining system responsiveness and focusing on the most important events.

Overview​

Configuration System​

Basic Configuration​

Context-Based Filtering​

Adaptive Sampling​

Quality-Based Filtering​

Basic Quality Metrics​

Custom Quality Calculation​

Performance Optimization​

Smart Sampling Strategies​

Batching for Performance​

State-Aware Illustrations​

Priority-Based Illustration​

Integration with Existing Code​

Migrating from Basic Illustrations​

ActionConfig Integration​

Monitoring and Debugging​

Performance Metrics​

Debugging Configuration​

Best Practices​

Configuration Strategy​

Performance Guidelines​

Quality Management​

Context Utilization​

Configuration Reference​

IllustrationConfig Properties​

BatchConfig Properties​

QualityMetrics Properties​