Modular Screen Capture System
Overviewโ
Brobot's screen capture system is designed to be completely modular, allowing you to switch between different capture providers (JAVACV_FFMPEG, Robot, FFmpeg, SikuliX) with just a single configuration property. This architecture provides maximum flexibility while maintaining a consistent API across all providers.
Default Configuration: Brobot uses JAVACV_FFMPEG as the default provider for 100% accurate physical resolution capture. No configuration needed for optimal pattern matching!
For comprehensive information on DPI scaling and resolution strategies, see the DPI and Resolution Guide.
Key Featuresโ
- Property-Based Configuration: Switch providers via
application.properties - Zero Code Changes: Change capture tools without modifying code
- Automatic Fallback: System selects best available provider
- DPI Scaling Support: Physical resolution capture with DPI disable strategy
- Unified Interface: Same API regardless of provider
- 100% Match Accuracy: Default configuration provides pixel-perfect pattern matching
Quick Startโ
Basic Configurationโ
The default configuration uses JAVACV_FFMPEG for optimal physical resolution capture:
# Default settings in brobot-defaults.properties:
brobot.capture.provider=JAVACV_FFMPEG # JavaCV FFmpeg for 100% pattern match accuracy
brobot.dpi.disable=true # Disable DPI awareness for physical resolution
brobot.dpi.resize-factor=1.0 # No pattern scaling (1:1 pixel matching)
No configuration needed! These defaults provide:
- Physical resolution capture (1920ร1080 on Full HD displays)
- 100% pattern match accuracy (pixel-perfect matching)
- No scaling artifacts or interpolation blur
- Works with patterns from any source (SikuliX IDE, Snipping Tool, etc.)
Available Providersโ
| Provider | Property Value | Dependencies | Best For |
|---|---|---|---|
| JAVACV_FFMPEG | JAVACV_FFMPEG | JavaCV (bundled) | Default - 100% accurate physical capture |
| Robot | ROBOT | None (built-in Java) | Fast, dynamic DPI scaling |
| FFmpeg | FFMPEG | External FFmpeg binary | Physical capture with external FFmpeg |
| SikuliX | SIKULIX | SikuliX library (included) | Legacy compatibility (Java version dependent) |
| Auto | AUTO | None | Automatic selection |
Provider Detailsโ
JAVACV_FFMPEG Provider (Default) โ โ
JAVACV_FFMPEG is the default and recommended provider for maximum pattern matching accuracy.
Advantages:
- 100% pattern match accuracy - Pixel-perfect matching with no scaling artifacts
- Physical resolution capture - Always captures at true hardware resolution (e.g., 1920ร1080)
- No external dependencies - JavaCV libraries bundled with Brobot
- Universal pattern compatibility - Works with patterns from any source
- Fast performance - Native capture with no runtime scaling overhead
- Cross-platform - Windows, macOS, Linux support
Configuration:
brobot.capture.provider=JAVACV_FFMPEG # Default - no configuration needed
brobot.dpi.disable=true # Physical resolution capture
brobot.dpi.resize-factor=1.0 # No pattern scaling
How It Works:
- Uses bundled JavaCV/FFmpeg libraries for native screen capture
- Disables Java's DPI awareness to capture at physical resolution
- No pattern scaling needed (1:1 pixel matching)
- Platform-specific optimizations (gdigrab on Windows, avfoundation on macOS, x11grab on Linux)
When to Use:
- โ Recommended for all new projects - Best accuracy and performance
- โ When you need 100% pattern match reliability
- โ When using patterns from multiple sources
- โ Production environments where accuracy is critical
SikuliX Providerโ
SikuliX is a legacy provider maintained for backward compatibility.
Advantages:
- Compatibility with older Brobot projects
- Works with existing SikuliX workflows
- No additional configuration for basic usage
Disadvantages:
- โ ๏ธ Java version dependent resolution behavior
- โ ๏ธ Lower pattern match accuracy (~77%) compared to physical capture
- โ ๏ธ May require pattern scaling with
resize-factor=auto
Configuration:
brobot.capture.provider=SIKULIX
brobot.dpi.disable=false # Enable DPI awareness
brobot.dpi.resize-factor=auto # Automatic pattern scaling
Resolution Behavior:
- Java 8: Captures at physical resolution
- Java 21+: Captures at logical resolution (requires resize-factor=auto for scaling)
When to Use:
- Migrating from older Brobot versions
- Existing projects using SikuliX patterns with auto-scaling
- Testing backward compatibility
Robot Providerโ
The Robot provider uses Java's built-in java.awt.Robot class with intelligent DPI scaling compensation.
Advantages:
- No external dependencies
- Fast in-memory operations
- Automatic DPI scaling to physical resolution
- Cross-platform support
Configuration:
brobot.capture.provider=ROBOT
brobot.capture.robot.scale-to-physical=true
brobot.capture.robot.expected-physical-width=1920
brobot.capture.robot.expected-physical-height=1080
DPI Scaling Detection: The Robot provider automatically detects Windows DPI scaling and compensates:
- 125% scaling: 1536x864 โ 1920x1080
- 150% scaling: 1280x720 โ 1920x1080
- 200% scaling: 960x540 โ 1920x1080
FFmpeg Provider (External)โ
The FFmpeg provider uses an external FFmpeg installation for screen capture. This is different from JAVACV_FFMPEG, which uses bundled libraries.
Advantages:
- True physical resolution capture
- Professional-grade quality
- Platform-specific optimizations
- No scaling artifacts
- Can use latest FFmpeg features
Disadvantages:
- โ Requires external FFmpeg installation (not bundled)
- โ ๏ธ Platform-specific setup required
- โ ๏ธ Version compatibility considerations
Configuration:
brobot.capture.provider=FFMPEG # Requires FFmpeg installed on system
brobot.capture.ffmpeg.timeout=5
brobot.capture.ffmpeg.format=png
brobot.capture.ffmpeg.log-level=error
Platform-Specific Capture Methods:
- Windows: Uses
gdigrab(requires FFmpeg.exe in PATH) - macOS: Uses
avfoundation(requires FFmpeg via Homebrew) - Linux: Uses
x11grab(requires FFmpeg package)
When to Use:
- You already have FFmpeg installed for other purposes
- You need specific FFmpeg features not in bundled JavaCV
- Otherwise, use JAVACV_FFMPEG instead (no installation needed)
Usage Examplesโ
Required Importsโ
// Core Java imports
import java.awt.Rectangle;
import java.awt.image.BufferedImage;
import java.io.IOException;
import java.util.Map;
// Spring Framework imports
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.stereotype.Service;
import jakarta.annotation.PostConstruct;
// Brobot imports
import io.github.jspinak.brobot.capture.UnifiedCaptureService;
import io.github.jspinak.brobot.capture.CaptureConfiguration;
import io.github.jspinak.brobot.capture.provider.CaptureProvider;
Basic Screen Captureโ
import io.github.jspinak.brobot.capture.UnifiedCaptureService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.awt.Rectangle;
import java.awt.image.BufferedImage;
import java.io.IOException;
@Service
public class ScreenCaptureService {
@Autowired
private UnifiedCaptureService captureService;
public BufferedImage captureFullScreen() throws IOException {
// Capture full screen
return captureService.captureScreen();
}
public BufferedImage captureMonitor(int screenId) throws IOException {
// Capture specific screen (multi-monitor)
return captureService.captureScreen(screenId);
}
public BufferedImage captureArea() throws IOException {
// Capture region
Rectangle region = new Rectangle(100, 100, 400, 300);
return captureService.captureRegion(region);
}
}
Runtime Provider Switchingโ
@Autowired
private CaptureConfiguration captureConfig;
// Switch providers at runtime
captureConfig.useRobot(); // Use Robot with scaling
captureConfig.useFFmpeg(); // Use FFmpeg (if available)
captureConfig.useSikuliX(); // Use SikuliX
captureConfig.useAuto(); // Automatic selection
// Check current configuration
String provider = captureConfig.getCurrentProvider();
boolean isPhysical = captureConfig.isCapturingPhysicalResolution();
Configuration Validationโ
// Validate configuration
if (captureConfig.validateConfiguration()) {
System.out.println("Capture configuration is valid");
}
// Get detailed configuration report
captureConfig.printConfigurationReport();
// Get all properties
Map<String, String> props = captureConfig.getAllCaptureProperties();
Configuration Referenceโ
Complete Properties Listโ
# ==================================================
# Main Capture Settings
# ==================================================
# Provider selection: JAVACV_FFMPEG, AUTO, ROBOT, FFMPEG, SIKULIX
# Default is JAVACV_FFMPEG for 100% accurate physical resolution capture
brobot.capture.provider=JAVACV_FFMPEG
# Prefer physical resolution captures
brobot.capture.prefer-physical=true
# Enable fallback to other providers
brobot.capture.fallback-enabled=true
# Enable debug logging
brobot.capture.enable-logging=false
# Retry configuration
brobot.capture.auto-retry=true
brobot.capture.retry-count=3
# ==================================================
# Robot Provider Settings
# ==================================================
# Scale to physical resolution
brobot.capture.robot.scale-to-physical=true
# Expected physical resolution
brobot.capture.robot.expected-physical-width=1920
brobot.capture.robot.expected-physical-height=1080
# ==================================================
# FFmpeg Provider Settings (uses bundled JavaCV)
# ==================================================
# Capture timeout (seconds)
brobot.capture.ffmpeg.timeout=5
# Output format
brobot.capture.ffmpeg.format=png
# Log level
brobot.capture.ffmpeg.log-level=error
Environment-Specific Profilesโ
# application-dev.properties
brobot.capture.provider=JAVACV_FFMPEG # Or use default
brobot.capture.enable-logging=true
# application-test.properties
brobot.capture.provider=AUTO # Flexible for CI/CD
brobot.capture.fallback-enabled=true
# application-prod.properties
brobot.capture.provider=JAVACV_FFMPEG # 100% accuracy
brobot.capture.retry-count=5
Advanced Topicsโ
Custom Provider Implementationโ
You can create custom capture providers by implementing the CaptureProvider interface:
@Component
public class CustomCaptureProvider implements CaptureProvider {
@Override
public BufferedImage captureScreen() throws IOException {
// Your implementation
}
@Override
public boolean isAvailable() {
// Check if your provider can work
}
@Override
public String getName() {
return "Custom";
}
@Override
public ResolutionType getResolutionType() {
return ResolutionType.PHYSICAL;
}
}
Register as a Spring bean and use via properties:
brobot.capture.provider=CUSTOM
Provider Selection Logicโ
When AUTO is configured, the selection order is:
- JAVACV_FFMPEG - Preferred (100% accuracy, bundled)
- Robot - Fallback (always available, dynamic DPI)
- FFmpeg - If external FFmpeg is installed
- SikuliX - Legacy fallback option
Recommendation: Use explicit brobot.capture.provider=JAVACV_FFMPEG instead of AUTO for predictable behavior.
Handling DPI Scalingโ
The system automatically handles DPI scaling in different ways:
- JAVACV_FFMPEG Provider (Default): Captures at true physical resolution with DPI awareness disabled
- Robot Provider: Detects and compensates via dynamic image scaling
- FFmpeg Provider: Captures at true physical resolution (requires external FFmpeg)
- SikuliX Provider: Behavior varies by Java version (8: physical, 21: logical)
For detailed DPI handling strategies and troubleshooting, see the DPI and Resolution Guide.
Performance Considerationsโ
| Provider | Speed | Memory Usage | Quality | Accuracy |
|---|---|---|---|---|
| JAVACV_FFMPEG | Fast | Low-Medium | Excellent | 100% |
| Robot | Fast | Low | Excellent (with scaling) | 95%+ |
| FFmpeg | Medium | Medium | Excellent | 100% |
| SikuliX | Medium | Medium | Variable | 77% |
Recommendation: JAVACV_FFMPEG provides the best balance of speed, quality, and accuracy.
Troubleshootingโ
Common Issuesโ
Provider Not Available
Error: Provider not available: FFMPEG
Solution:
- If using external
FFMPEG: Install FFmpeg binary on your system - Recommended: Switch to
JAVACV_FFMPEG(no installation needed) - Alternative: Switch to
ROBOTprovider (always available)
Wrong Resolution Captured
Captured: 1536x864 (expected 1920x1080)
Diagnosis: This indicates logical resolution capture (DPI scaling issue).
Solution 1 (Recommended): Use default JAVACV_FFMPEG provider:
brobot.capture.provider=JAVACV_FFMPEG
brobot.dpi.disable=true
brobot.dpi.resize-factor=1.0
Solution 2: Enable Robot physical scaling:
brobot.capture.provider=ROBOT
brobot.capture.robot.scale-to-physical=true
brobot.capture.robot.expected-physical-width=1920
brobot.capture.robot.expected-physical-height=1080
See the DPI and Resolution Guide for comprehensive troubleshooting.
Capture Fails Intermittently Solution: Enable retry logic:
brobot.capture.auto-retry=true
brobot.capture.retry-count=5
Debuggingโ
Enable logging to diagnose issues:
brobot.capture.enable-logging=true
Check provider status programmatically:
System.out.println(captureService.getProvidersInfo());
Migration Guideโ
From Direct SikuliX Usageโ
Before:
Screen screen = new Screen();
BufferedImage img = screen.capture().getImage();
After:
@Autowired
private UnifiedCaptureService captureService;
BufferedImage img = captureService.captureScreen();
From Direct Robot Usageโ
Before:
Robot robot = new Robot();
BufferedImage img = robot.createScreenCapture(bounds);
After:
@Autowired
private UnifiedCaptureService captureService;
BufferedImage img = captureService.captureRegion(bounds);
Best Practicesโ
-
Use Properties for Configuration
- Configure via properties files, not code
- Use Spring profiles for different environments
-
Enable Fallback for Production
brobot.capture.fallback-enabled=true -
Validate Configuration on Startup
@PostConstruct
public void validateCapture() {
captureConfig.validateConfiguration();
} -
Monitor Provider Status
- Log provider selection
- Monitor capture failures
- Track performance metrics
-
Choose Appropriate Provider
- Recommended (Default):
JAVACV_FFMPEG(100% accuracy, no setup) - Development:
JAVACV_FFMPEGorROBOT(no setup required) - CI/CD:
AUTO(flexible) orJAVACV_FFMPEG(reliable) - Production:
JAVACV_FFMPEG(accuracy + performance) orROBOT(fallback)
- Recommended (Default):
API Referenceโ
UnifiedCaptureServiceโ
Primary service for all capture operations:
@Service
@Primary
public class UnifiedCaptureService {
// Capture methods
public BufferedImage captureScreen() throws IOException;
public BufferedImage captureScreen(int screenId) throws IOException;
public BufferedImage captureRegion(Rectangle region) throws IOException;
public BufferedImage captureRegion(int screenId, Rectangle region) throws IOException;
// Provider management
public void setProvider(String providerName);
public CaptureProvider getActiveProvider();
public String getActiveProviderName();
public String getProvidersInfo();
public boolean isPhysicalResolution();
}
CaptureConfigurationโ
Configuration and management helper:
@Component
public class CaptureConfiguration {
// Provider switching
public void useRobot();
public void useFFmpeg();
public void useSikuliX();
public void useAuto();
public void setCaptureMode(CaptureMode mode);
// Configuration inspection
public String getCurrentProvider();
public boolean isCapturingPhysicalResolution();
public boolean validateConfiguration();
public Map<String, String> getAllCaptureProperties();
public void printConfigurationReport();
}
Summaryโ
The modular capture system provides:
- Complete Flexibility: Switch providers with a single property
- Zero Code Impact: No code changes when switching
- Production Ready: Automatic fallback and retry logic
- DPI Aware: Handles Windows scaling correctly
- Extensible: Support for custom providers
Simply set brobot.capture.provider in your properties file and let the system handle the rest!
Related Documentationโ
Core Capture Documentationโ
- Capture Quick Reference - Quick provider switching guide with decision tree and common scenarios
- DPI and Resolution Guide - Comprehensive DPI scaling and resolution handling strategies
- Capture Methods Comparison - Detailed performance benchmarks comparing all providers
Configurationโ
- Configuration Properties Reference - Complete reference for all Brobot configuration properties
- Auto-Configuration - Spring Boot auto-configuration and integration details
- Headless Configuration - Configuring Brobot for CI/CD and headless environments
Testing & CI/CDโ
- Testing Introduction - Overview of Brobot testing strategies and patterns
- Mock Mode Guide - Testing without screen interaction using mock mode
- Integration Testing - Spring Boot integration testing with Brobot
- CI/CD Testing Guide - Testing Brobot applications in CI/CD pipelines
Getting Startedโ
- Introduction - Brobot overview, why use Brobot, and core concepts
- Installation - Adding Brobot dependencies to your project
- Quick Start - Get started with Brobot quickly