Skip to main content
Version: Latest

Quick Start: Pattern Capture Setup

This guide helps you quickly set up the optimal pattern capture workflow based on extensive testing and comparison.

TL;DR - Best Configuration​

For complete configuration details, see the Properties Reference.

# Add to application.properties
brobot.capture.provider=JAVACV_FFMPEG
brobot.dpi.resize-factor=auto
brobot.action.similarity=0.70

For pattern creation, use OS native tools:

  • Windows: Windows Snipping Tool (Win+Shift+S) - 95-100% match rate
  • macOS: Built-in screenshot (Cmd+Shift+4) - 95-100% match rate
  • Linux: GNOME Screenshot or Spectacle - 95-100% match rate

Note: While FFmpeg captures match Windows screenshots pixel-perfectly, Windows Snipping Tool patterns achieve better runtime match rates. For detailed comparison, see the Capture Methods Comparison and Pattern Creation Tools Guide.

Why This Configuration?​

Based on comprehensive testing comparing 8 different capture methods:

  • Windows Snipping Tool patterns achieve 95-100% runtime match rates
  • FFmpeg patterns only achieve 70-80% runtime match rates
  • Clean, artifact-free patterns from native OS tools match better
  • JavaCV FFmpeg for runtime capture at physical resolution (1920x1080)

For the complete technical explanation, see the Modular Capture System documentation.

Setup Instructions​

Step 1: Configure Brobot​

Add to your application.properties:

# Essential settings for optimal capture
brobot.capture.provider=JAVACV_FFMPEG  # Use bundled FFmpeg
brobot.dpi.resize-factor=auto          # Auto-detect screen scaling
brobot.capture.prefer-physical=true    # Prefer physical resolution

# Optional: Fine-tuning
brobot.action.similarity=0.70          # Can go higher with FFmpeg
brobot.capture.enable-logging=true     # See what's happening

Step 2: Choose Your Capture Tool​

Windows:

  1. Press Win+Shift+S
  2. Select the UI element
  3. Save to your project's images/[state-name]/ folder
  4. Name descriptively (e.g., login-button.png)

macOS:

  1. Press Cmd+Shift+4
  2. Select the UI element
  3. Save as PNG

Linux:

  1. Use GNOME Screenshot or Spectacle
  2. Select region mode
  3. Save as PNG

Pros: Best runtime match rates (95-100%), no setup required Cons: Manual file organization

Option B: Brobot Pattern Capture Tool (For Testing)​

For complete usage instructions, see the Pattern Capture Tool Guide.

  1. Run the tool: java -jar pattern-capture-tool-1.0.0.jar
  2. Use any provider for testing similarity
  3. Press F1 or click "Capture"
  4. Auto-saves to patterns/ folder with timestamp

Pros: Good for testing patterns, auto-organization Cons: Lower runtime match rates (70-80%) than native OS tools

Option C: SikuliX IDE (For Testing Only)​

SikuliX IDE can be used to test similarity thresholds but achieves lower runtime match rates (70-80%) compared to native OS tools.

Verification​

Test Your Setup​

For comprehensive testing strategies, see the Testing Introduction and Mock Mode Guide.

  1. Capture a test pattern using your chosen method
  2. Run this verification code:
package io.github.jspinak.brobot.example;

import static org.junit.jupiter.api.Assertions.*;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import io.github.jspinak.brobot.action.Action;
import io.github.jspinak.brobot.action.ActionResult;
import io.github.jspinak.brobot.model.match.Match;
import io.github.jspinak.brobot.model.state.StateImage;
import io.github.jspinak.brobot.test.BrobotTestBase;

/**
 * Test to verify pattern capture setup.
 */
@Component
public class VerifyPatternCaptureTest extends BrobotTestBase {

    @Autowired
    private Action action;

    @Test
    public void verifyPatternCapture() {
        // Your captured pattern - use addPattern(), not withPath()
        StateImage testPattern = new StateImage.Builder()
            .addPattern("images/test/my-pattern.png")
            .build();

        // Should find with high confidence
        ActionResult result = action.find(testPattern);
        assertTrue(result.isSuccess());

        // Get similarity score from best match
        if (result.getBestMatch().isPresent()) {
            Match bestMatch = result.getBestMatch().get();
            assertTrue(bestMatch.getScore() > 0.90,
                      "Score should be >90% with FFmpeg: " + bestMatch.getScore());
        } else {
            fail("No match found");
        }
    }
}

Expected Results​

With correct setup:

  • Similarity scores > 90% for exact matches
  • Consistent results across different capture sessions
  • Fast pattern matching (< 1 second for full screen)

Common Patterns by Resolution​

Your patterns will be captured at:

Display ScalingCapture ResolutionMethod
100% (No scaling)1920x1080All methods same
125% (Recommended)1920x1080FFmpeg, Windows
125% (Recommended)1536x864SikuliX, Robot
150%1920x1080FFmpeg, Windows
150%1280x720SikuliX, Robot

FFmpeg always captures at physical resolution, making patterns portable across different scaling settings. For detailed DPI handling strategies, see the DPI Resolution Guide.

Quick Troubleshooting​

For comprehensive debugging strategies, see the Debugging Pattern Matching guide.

"Pattern not found" with high threshold​

# Lower similarity temporarily to debug
brobot.action.similarity=0.60
brobot.console.actions.enabled=true  # See match scores

"FFmpeg not available"​

package io.github.jspinak.brobot.example;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import io.github.jspinak.brobot.capture.CaptureConfiguration;

/**
 * Debugging helper to check available capture providers.
 */
@Component
public class CaptureProviderCheck {

    @Autowired
    private CaptureConfiguration captureConfig;

    public void checkAvailableProviders() {
        // Check available providers - correct method name is printConfigurationReport()
        captureConfig.printConfigurationReport();
    }
}

Different colors between captures​

This is normal and doesn't affect matching. FFmpeg handles color spaces consistently.

Best Practices​

DO ✅​

  • Use FFmpeg provider for new projects
  • Capture at physical resolution when possible
  • Keep patterns small and focused (< 200x200 pixels)
  • Name patterns descriptively

DON'T ❌​

  • Mix patterns from different providers without testing
  • Capture entire windows (use specific UI elements)
  • Use similarity threshold below 0.60
  • Ignore DPI scaling on high-DPI displays

Performance Tips​

Based on testing with 1000+ patterns:

  1. Small patterns match faster

    • 50x50: ~0.1s
    • 200x200: ~0.3s
    • Full screen: ~1.0s

  2. FFmpeg has best compression

    • 20-30% smaller files than other methods
    • Faster loading from disk

  3. Batch capture for efficiency

    • Use Pattern Capture Tool for multiple patterns
    • Maintains consistent capture settings

Note: Performance metrics are approximate and vary based on system configuration, screen resolution, image complexity, and CPU load. Use these as general guidelines for relative performance comparison, not exact measurements.

Next Steps​

  1. ✅ Configure Brobot with recommended settings
  2. ✅ Choose your capture tool (Windows or Brobot Tool)
  3. ✅ Capture a test pattern and verify
  4. 📖 Read full comparison study for detailed analysis
  5. 🚀 Start capturing patterns for your automation!

Summary​

The extensive testing proved that FFmpeg runtime capture with Windows Snipping Tool patterns provides optimal results. While the Brobot Pattern Capture Tool is useful for testing similarity thresholds, Windows Snipping Tool patterns achieve significantly better runtime match rates (95-100% vs 70-80%). This configuration ensures:

  • Clean, artifact-free patterns from native OS tools
  • Best runtime match rates (95-100%)
  • Consistent pattern matching across sessions
  • Optimal file compression with FFmpeg
  • Resolution independence

Key Finding: Pattern creation tool matters more than you might expect. Use Windows Snipping Tool (or macOS/Linux equivalents) for patterns, and FFmpeg for runtime capture. This combination is proven by extensive testing across 1000+ patterns.

No more guessing - this configuration is proven by data!

Pattern Capture & Creation​

Capture System Architecture​

Configuration & Properties​

Testing & Debugging​

Getting Started​