Skip to main content
Version: Latest

Modular Screen Capture System

Overviewโ€‹

Brobot's screen capture system is designed to be completely modular, allowing you to switch between different capture providers (JAVACV_FFMPEG, Robot, FFmpeg, SikuliX) with just a single configuration property. This architecture provides maximum flexibility while maintaining a consistent API across all providers.

Default Configuration: Brobot uses JAVACV_FFMPEG as the default provider for 100% accurate physical resolution capture. No configuration needed for optimal pattern matching!

For comprehensive information on DPI scaling and resolution strategies, see the DPI and Resolution Guide.

Key Featuresโ€‹

  • Property-Based Configuration: Switch providers via application.properties
  • Zero Code Changes: Change capture tools without modifying code
  • Automatic Fallback: System selects best available provider
  • DPI Scaling Support: Physical resolution capture with DPI disable strategy
  • Unified Interface: Same API regardless of provider
  • 100% Match Accuracy: Default configuration provides pixel-perfect pattern matching

Quick Startโ€‹

Basic Configurationโ€‹

The default configuration uses JAVACV_FFMPEG for optimal physical resolution capture:

# Default settings in brobot-defaults.properties:
brobot.capture.provider=JAVACV_FFMPEG # JavaCV FFmpeg for 100% pattern match accuracy
brobot.dpi.disable=true # Disable DPI awareness for physical resolution
brobot.dpi.resize-factor=1.0 # No pattern scaling (1:1 pixel matching)

No configuration needed! These defaults provide:

  • Physical resolution capture (1920ร—1080 on Full HD displays)
  • 100% pattern match accuracy (pixel-perfect matching)
  • No scaling artifacts or interpolation blur
  • Works with patterns from any source (SikuliX IDE, Snipping Tool, etc.)

Available Providersโ€‹

ProviderProperty ValueDependenciesBest For
JAVACV_FFMPEGJAVACV_FFMPEGJavaCV (bundled)Default - 100% accurate physical capture
RobotROBOTNone (built-in Java)Fast, dynamic DPI scaling
FFmpegFFMPEGExternal FFmpeg binaryPhysical capture with external FFmpeg
SikuliXSIKULIXSikuliX library (included)Legacy compatibility (Java version dependent)
AutoAUTONoneAutomatic selection

Provider Detailsโ€‹

JAVACV_FFMPEG Provider (Default) โœ…โ€‹

JAVACV_FFMPEG is the default and recommended provider for maximum pattern matching accuracy.

Advantages:

  • 100% pattern match accuracy - Pixel-perfect matching with no scaling artifacts
  • Physical resolution capture - Always captures at true hardware resolution (e.g., 1920ร—1080)
  • No external dependencies - JavaCV libraries bundled with Brobot
  • Universal pattern compatibility - Works with patterns from any source
  • Fast performance - Native capture with no runtime scaling overhead
  • Cross-platform - Windows, macOS, Linux support

Configuration:

brobot.capture.provider=JAVACV_FFMPEG  # Default - no configuration needed
brobot.dpi.disable=true # Physical resolution capture
brobot.dpi.resize-factor=1.0 # No pattern scaling

How It Works:

  • Uses bundled JavaCV/FFmpeg libraries for native screen capture
  • Disables Java's DPI awareness to capture at physical resolution
  • No pattern scaling needed (1:1 pixel matching)
  • Platform-specific optimizations (gdigrab on Windows, avfoundation on macOS, x11grab on Linux)

When to Use:

  • โœ… Recommended for all new projects - Best accuracy and performance
  • โœ… When you need 100% pattern match reliability
  • โœ… When using patterns from multiple sources
  • โœ… Production environments where accuracy is critical

SikuliX Providerโ€‹

SikuliX is a legacy provider maintained for backward compatibility.

Advantages:

  • Compatibility with older Brobot projects
  • Works with existing SikuliX workflows
  • No additional configuration for basic usage

Disadvantages:

  • โš ๏ธ Java version dependent resolution behavior
  • โš ๏ธ Lower pattern match accuracy (~77%) compared to physical capture
  • โš ๏ธ May require pattern scaling with resize-factor=auto

Configuration:

brobot.capture.provider=SIKULIX
brobot.dpi.disable=false # Enable DPI awareness
brobot.dpi.resize-factor=auto # Automatic pattern scaling

Resolution Behavior:

  • Java 8: Captures at physical resolution
  • Java 21+: Captures at logical resolution (requires resize-factor=auto for scaling)

When to Use:

  • Migrating from older Brobot versions
  • Existing projects using SikuliX patterns with auto-scaling
  • Testing backward compatibility

Robot Providerโ€‹

The Robot provider uses Java's built-in java.awt.Robot class with intelligent DPI scaling compensation.

Advantages:

  • No external dependencies
  • Fast in-memory operations
  • Automatic DPI scaling to physical resolution
  • Cross-platform support

Configuration:

brobot.capture.provider=ROBOT
brobot.capture.robot.scale-to-physical=true
brobot.capture.robot.expected-physical-width=1920
brobot.capture.robot.expected-physical-height=1080

DPI Scaling Detection: The Robot provider automatically detects Windows DPI scaling and compensates:

  • 125% scaling: 1536x864 โ†’ 1920x1080
  • 150% scaling: 1280x720 โ†’ 1920x1080
  • 200% scaling: 960x540 โ†’ 1920x1080

FFmpeg Provider (External)โ€‹

The FFmpeg provider uses an external FFmpeg installation for screen capture. This is different from JAVACV_FFMPEG, which uses bundled libraries.

Advantages:

  • True physical resolution capture
  • Professional-grade quality
  • Platform-specific optimizations
  • No scaling artifacts
  • Can use latest FFmpeg features

Disadvantages:

  • โŒ Requires external FFmpeg installation (not bundled)
  • โš ๏ธ Platform-specific setup required
  • โš ๏ธ Version compatibility considerations

Configuration:

brobot.capture.provider=FFMPEG  # Requires FFmpeg installed on system
brobot.capture.ffmpeg.timeout=5
brobot.capture.ffmpeg.format=png
brobot.capture.ffmpeg.log-level=error

Platform-Specific Capture Methods:

  • Windows: Uses gdigrab (requires FFmpeg.exe in PATH)
  • macOS: Uses avfoundation (requires FFmpeg via Homebrew)
  • Linux: Uses x11grab (requires FFmpeg package)

When to Use:

  • You already have FFmpeg installed for other purposes
  • You need specific FFmpeg features not in bundled JavaCV
  • Otherwise, use JAVACV_FFMPEG instead (no installation needed)

Usage Examplesโ€‹

Required Importsโ€‹

// Core Java imports
import java.awt.Rectangle;
import java.awt.image.BufferedImage;
import java.io.IOException;
import java.util.Map;

// Spring Framework imports
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.stereotype.Service;
import jakarta.annotation.PostConstruct;

// Brobot imports
import io.github.jspinak.brobot.capture.UnifiedCaptureService;
import io.github.jspinak.brobot.capture.CaptureConfiguration;
import io.github.jspinak.brobot.capture.provider.CaptureProvider;

Basic Screen Captureโ€‹

import io.github.jspinak.brobot.capture.UnifiedCaptureService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.awt.Rectangle;
import java.awt.image.BufferedImage;
import java.io.IOException;

@Service
public class ScreenCaptureService {

@Autowired
private UnifiedCaptureService captureService;

public BufferedImage captureFullScreen() throws IOException {
// Capture full screen
return captureService.captureScreen();
}

public BufferedImage captureMonitor(int screenId) throws IOException {
// Capture specific screen (multi-monitor)
return captureService.captureScreen(screenId);
}

public BufferedImage captureArea() throws IOException {
// Capture region
Rectangle region = new Rectangle(100, 100, 400, 300);
return captureService.captureRegion(region);
}
}

Runtime Provider Switchingโ€‹

@Autowired
private CaptureConfiguration captureConfig;

// Switch providers at runtime
captureConfig.useRobot(); // Use Robot with scaling
captureConfig.useFFmpeg(); // Use FFmpeg (if available)
captureConfig.useSikuliX(); // Use SikuliX
captureConfig.useAuto(); // Automatic selection

// Check current configuration
String provider = captureConfig.getCurrentProvider();
boolean isPhysical = captureConfig.isCapturingPhysicalResolution();

Configuration Validationโ€‹

// Validate configuration
if (captureConfig.validateConfiguration()) {
System.out.println("Capture configuration is valid");
}

// Get detailed configuration report
captureConfig.printConfigurationReport();

// Get all properties
Map<String, String> props = captureConfig.getAllCaptureProperties();

Configuration Referenceโ€‹

Complete Properties Listโ€‹

# ==================================================
# Main Capture Settings
# ==================================================
# Provider selection: JAVACV_FFMPEG, AUTO, ROBOT, FFMPEG, SIKULIX
# Default is JAVACV_FFMPEG for 100% accurate physical resolution capture
brobot.capture.provider=JAVACV_FFMPEG

# Prefer physical resolution captures
brobot.capture.prefer-physical=true

# Enable fallback to other providers
brobot.capture.fallback-enabled=true

# Enable debug logging
brobot.capture.enable-logging=false

# Retry configuration
brobot.capture.auto-retry=true
brobot.capture.retry-count=3

# ==================================================
# Robot Provider Settings
# ==================================================
# Scale to physical resolution
brobot.capture.robot.scale-to-physical=true

# Expected physical resolution
brobot.capture.robot.expected-physical-width=1920
brobot.capture.robot.expected-physical-height=1080

# ==================================================
# FFmpeg Provider Settings (uses bundled JavaCV)
# ==================================================
# Capture timeout (seconds)
brobot.capture.ffmpeg.timeout=5

# Output format
brobot.capture.ffmpeg.format=png

# Log level
brobot.capture.ffmpeg.log-level=error

Environment-Specific Profilesโ€‹

# application-dev.properties
brobot.capture.provider=JAVACV_FFMPEG # Or use default
brobot.capture.enable-logging=true

# application-test.properties
brobot.capture.provider=AUTO # Flexible for CI/CD
brobot.capture.fallback-enabled=true

# application-prod.properties
brobot.capture.provider=JAVACV_FFMPEG # 100% accuracy
brobot.capture.retry-count=5

Advanced Topicsโ€‹

Custom Provider Implementationโ€‹

You can create custom capture providers by implementing the CaptureProvider interface:

@Component
public class CustomCaptureProvider implements CaptureProvider {

@Override
public BufferedImage captureScreen() throws IOException {
// Your implementation
}

@Override
public boolean isAvailable() {
// Check if your provider can work
}

@Override
public String getName() {
return "Custom";
}

@Override
public ResolutionType getResolutionType() {
return ResolutionType.PHYSICAL;
}
}

Register as a Spring bean and use via properties:

brobot.capture.provider=CUSTOM

Provider Selection Logicโ€‹

When AUTO is configured, the selection order is:

  1. JAVACV_FFMPEG - Preferred (100% accuracy, bundled)
  2. Robot - Fallback (always available, dynamic DPI)
  3. FFmpeg - If external FFmpeg is installed
  4. SikuliX - Legacy fallback option

Recommendation: Use explicit brobot.capture.provider=JAVACV_FFMPEG instead of AUTO for predictable behavior.

Handling DPI Scalingโ€‹

The system automatically handles DPI scaling in different ways:

  • JAVACV_FFMPEG Provider (Default): Captures at true physical resolution with DPI awareness disabled
  • Robot Provider: Detects and compensates via dynamic image scaling
  • FFmpeg Provider: Captures at true physical resolution (requires external FFmpeg)
  • SikuliX Provider: Behavior varies by Java version (8: physical, 21: logical)

For detailed DPI handling strategies and troubleshooting, see the DPI and Resolution Guide.

Performance Considerationsโ€‹

ProviderSpeedMemory UsageQualityAccuracy
JAVACV_FFMPEGFastLow-MediumExcellent100%
RobotFastLowExcellent (with scaling)95%+
FFmpegMediumMediumExcellent100%
SikuliXMediumMediumVariable77%

Recommendation: JAVACV_FFMPEG provides the best balance of speed, quality, and accuracy.

Troubleshootingโ€‹

Common Issuesโ€‹

Provider Not Available

Error: Provider not available: FFMPEG

Solution:

  • If using external FFMPEG: Install FFmpeg binary on your system
  • Recommended: Switch to JAVACV_FFMPEG (no installation needed)
  • Alternative: Switch to ROBOT provider (always available)

Wrong Resolution Captured

Captured: 1536x864 (expected 1920x1080)

Diagnosis: This indicates logical resolution capture (DPI scaling issue).

Solution 1 (Recommended): Use default JAVACV_FFMPEG provider:

brobot.capture.provider=JAVACV_FFMPEG
brobot.dpi.disable=true
brobot.dpi.resize-factor=1.0

Solution 2: Enable Robot physical scaling:

brobot.capture.provider=ROBOT
brobot.capture.robot.scale-to-physical=true
brobot.capture.robot.expected-physical-width=1920
brobot.capture.robot.expected-physical-height=1080

See the DPI and Resolution Guide for comprehensive troubleshooting.

Capture Fails Intermittently Solution: Enable retry logic:

brobot.capture.auto-retry=true
brobot.capture.retry-count=5

Debuggingโ€‹

Enable logging to diagnose issues:

brobot.capture.enable-logging=true

Check provider status programmatically:

System.out.println(captureService.getProvidersInfo());

Migration Guideโ€‹

From Direct SikuliX Usageโ€‹

Before:

Screen screen = new Screen();
BufferedImage img = screen.capture().getImage();

After:

@Autowired
private UnifiedCaptureService captureService;

BufferedImage img = captureService.captureScreen();

From Direct Robot Usageโ€‹

Before:

Robot robot = new Robot();
BufferedImage img = robot.createScreenCapture(bounds);

After:

@Autowired
private UnifiedCaptureService captureService;

BufferedImage img = captureService.captureRegion(bounds);

Best Practicesโ€‹

  1. Use Properties for Configuration

    • Configure via properties files, not code
    • Use Spring profiles for different environments
  2. Enable Fallback for Production

    brobot.capture.fallback-enabled=true
  3. Validate Configuration on Startup

    @PostConstruct
    public void validateCapture() {
    captureConfig.validateConfiguration();
    }
  4. Monitor Provider Status

    • Log provider selection
    • Monitor capture failures
    • Track performance metrics
  5. Choose Appropriate Provider

    • Recommended (Default): JAVACV_FFMPEG (100% accuracy, no setup)
    • Development: JAVACV_FFMPEG or ROBOT (no setup required)
    • CI/CD: AUTO (flexible) or JAVACV_FFMPEG (reliable)
    • Production: JAVACV_FFMPEG (accuracy + performance) or ROBOT (fallback)

API Referenceโ€‹

UnifiedCaptureServiceโ€‹

Primary service for all capture operations:

@Service
@Primary
public class UnifiedCaptureService {
// Capture methods
public BufferedImage captureScreen() throws IOException;
public BufferedImage captureScreen(int screenId) throws IOException;
public BufferedImage captureRegion(Rectangle region) throws IOException;
public BufferedImage captureRegion(int screenId, Rectangle region) throws IOException;

// Provider management
public void setProvider(String providerName);
public CaptureProvider getActiveProvider();
public String getActiveProviderName();
public String getProvidersInfo();
public boolean isPhysicalResolution();
}

CaptureConfigurationโ€‹

Configuration and management helper:

@Component
public class CaptureConfiguration {
// Provider switching
public void useRobot();
public void useFFmpeg();
public void useSikuliX();
public void useAuto();
public void setCaptureMode(CaptureMode mode);

// Configuration inspection
public String getCurrentProvider();
public boolean isCapturingPhysicalResolution();
public boolean validateConfiguration();
public Map<String, String> getAllCaptureProperties();
public void printConfigurationReport();
}

Summaryโ€‹

The modular capture system provides:

  • Complete Flexibility: Switch providers with a single property
  • Zero Code Impact: No code changes when switching
  • Production Ready: Automatic fallback and retry logic
  • DPI Aware: Handles Windows scaling correctly
  • Extensible: Support for custom providers

Simply set brobot.capture.provider in your properties file and let the system handle the rest!


Core Capture Documentationโ€‹

Configurationโ€‹

Testing & CI/CDโ€‹

Getting Startedโ€‹