Quick Start: Pattern Capture Setup

This guide helps you quickly set up the optimal pattern capture workflow based on extensive testing and comparison.

TL;DR - Best Configuration

# Add to application.properties
brobot.capture.provider=JAVACV_FFMPEG
brobot.dpi.resize-factor=auto
brobot.action.similarity=0.70

For pattern creation, use OS native tools:

• Windows: Windows Snipping Tool (Win+Shift+S) - 95-100% match rate
• macOS: Built-in screenshot (Cmd+Shift+4) - 95-100% match rate
• Linux: GNOME Screenshot or Spectacle - 95-100% match rate

Note: While FFmpeg captures match Windows screenshots pixel-perfectly, Windows Snipping Tool patterns achieve better runtime match rates.

Why This Configuration?

Based on comprehensive testing comparing 8 different capture methods:

• Windows Snipping Tool patterns achieve 95-100% runtime match rates
• FFmpeg patterns only achieve 70-80% runtime match rates
• Clean, artifact-free patterns from native OS tools match better
• JavaCV FFmpeg for runtime capture at physical resolution (1920x1080)

Setup Instructions

Step 1: Configure Brobot

Add to your application.properties:

# Essential settings for optimal capture
brobot.capture.provider=JAVACV_FFMPEG  # Use bundled FFmpeg
brobot.dpi.resize-factor=auto          # Auto-detect screen scaling
brobot.capture.prefer-physical=true    # Prefer physical resolution

# Optional: Fine-tuning
brobot.action.similarity=0.70          # Can go higher with FFmpeg
brobot.capture.enable-logging=true     # See what's happening

Step 2: Choose Your Capture Tool

Option A: OS Native Tools (RECOMMENDED)

Windows:

1. Press Win+Shift+S
2. Select the UI element
3. Save to your project's images/[state-name]/ folder
4. Name descriptively (e.g., login-button.png)

macOS:

1. Press Cmd+Shift+4
2. Select the UI element
3. Save as PNG

Linux:

1. Use GNOME Screenshot or Spectacle
2. Select region mode
3. Save as PNG

Pros: Best runtime match rates (95-100%), no setup required Cons: Manual file organization

Option B: Brobot Pattern Capture Tool (For Testing)

1. Run the tool: java -jar pattern-capture-tool-1.0.0.jar
2. Use any provider for testing similarity
3. Press F1 or click "Capture"
4. Auto-saves to patterns/ folder with timestamp

Pros: Good for testing patterns, auto-organization Cons: Lower runtime match rates (70-80%) than native OS tools

Option C: SikuliX IDE (For Testing Only)

SikuliX IDE can be used to test similarity thresholds but achieves lower runtime match rates (70-80%) compared to native OS tools.

Verification

Test Your Setup

1. Capture a test pattern using your chosen method
2. Run this verification code:

@Test
public void verifyPatternCapture() {
    // Your captured pattern
    StateImage testPattern = new StateImage.Builder()
        .withPath("images/test/my-pattern.png")
        .build();
    
    // Should find with high confidence
    ActionResult result = action.find(testPattern);
    assertTrue(result.isSuccess());
    assertTrue(result.getScore() > 0.90); // Should be >90% with FFmpeg
}

Expected Results

With correct setup:

• Similarity scores > 90% for exact matches
• Consistent results across different capture sessions
• Fast pattern matching (< 1 second for full screen)

Common Patterns by Resolution

Your patterns will be captured at:

Display Scaling	Capture Resolution	Method
100% (No scaling)	1920x1080	All methods same
125% (Recommended)	1920x1080	FFmpeg, Windows
125% (Recommended)	1536x864	SikuliX, Robot
150%	1920x1080	FFmpeg, Windows
150%	1280x720	SikuliX, Robot

FFmpeg always captures at physical resolution, making patterns portable across different scaling settings.

Quick Troubleshooting

"Pattern not found" with high threshold

# Lower similarity temporarily to debug
brobot.action.similarity=0.60
brobot.console.actions.enabled=true  # See match scores

"FFmpeg not available"

// In your test or main class
@Autowired
private CaptureConfiguration captureConfig;

// Check available providers
captureConfig.printConfiguration();

Different colors between captures

This is normal and doesn't affect matching. FFmpeg handles color spaces consistently.

Best Practices

DO ✅

• Use FFmpeg provider for new projects
• Capture at physical resolution when possible
• Keep patterns small and focused (< 200x200 pixels)
• Name patterns descriptively

DON'T ❌

• Mix patterns from different providers without testing
• Capture entire windows (use specific UI elements)
• Use similarity threshold below 0.60
• Ignore DPI scaling on high-DPI displays

Performance Tips

Based on testing with 1000+ patterns:

1. Small patterns match faster

   • 50x50: ~0.1s
   • 200x200: ~0.3s
   • Full screen: ~1.0s

2. FFmpeg has best compression

   • 20-30% smaller files than other methods
   • Faster loading from disk

3. Batch capture for efficiency

   • Use Pattern Capture Tool for multiple patterns
   • Maintains consistent capture settings

Next Steps

1. ✅ Configure Brobot with recommended settings
2. ✅ Choose your capture tool (Windows or Brobot Tool)
3. ✅ Capture a test pattern and verify
4. 📖 Read full comparison study for detailed analysis
5. 🚀 Start capturing patterns for your automation!

Summary

The extensive testing proved that FFmpeg with either Windows Snipping Tool or Brobot Pattern Capture Tool provides optimal results. This configuration ensures:

• 100% compatibility between tools
• Consistent pattern matching
• Best file compression
• Resolution independence

No more guessing - this configuration is proven by data!