Getting Started with MCP Server
The Model Context Protocol (MCP) Server enables AI agents to control Brobot automations through a RESTful API. This guide will help you set up and start using the MCP server in minutes.
What is MCP Server?โ
The MCP Server acts as a bridge between AI systems and Brobot's automation capabilities:
AI Agent (GPT-4, Claude) โ MCP Server โ Brobot โ Your Application
This enables:
- ๐ค Natural Language Control: Tell AI what to do in plain English
- ๐๏ธ Visual Feedback: AI can see screenshots and make decisions
- ๐ State-Based Automation: Leverage Brobot's reliable state management
- ๐ Parallel Processing: Handle multiple automation requests
Prerequisitesโ
Before starting, ensure you have:
- โ Python 3.8 or higher
- โ Java 11 or higher
- โ Brobot framework installed
- โ A GUI application to automate
Quick Startโ
1. Install MCP Serverโ
# Clone the repository
git clone https://github.com/jspinak/brobot-mcp-server.git
cd brobot-mcp-server
# Install Python dependencies
pip install -e .
2. Build the CLI Bridgeโ
# Build the Java CLI that connects to Brobot
cd brobot-cli
gradle shadowJar
cd ..
3. Configure the Serverโ
Create a .env
file:
# Disable mock mode to use real Brobot
USE_MOCK_DATA=false
# Path to the CLI JAR
BROBOT_CLI_JAR=brobot-cli/build/libs/brobot-cli.jar
4. Start the Serverโ
python -m mcp_server.main
The server is now running at http://localhost:8000
!
Visit http://localhost:8000/docs
for interactive API documentation.
Your First AI Automationโ
Using Python Clientโ
Install the client library:
pip install brobot-client
Create a simple automation:
from brobot_client import BrobotClient
import openai
# Initialize clients
brobot = BrobotClient()
openai.api_key = "your-api-key"
# Get current screen state
observation = brobot.get_observation()
# Ask AI what to do
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[{
"role": "user",
"content": f"I see these UI elements: {observation.active_states}. How do I log in?"
}]
)
# AI might respond: "Click on the login button, enter credentials, then submit"
# Execute the suggested actions
brobot.click("login_button.png")
brobot.type_text("username@example.com")
brobot.click("submit_button.png")
Using Direct API Callsโ
import requests
# Get current observation
response = requests.get("http://localhost:8000/api/v1/observation")
observation = response.json()
# Execute an action
action = {
"action_type": "click",
"parameters": {
"image_pattern": "login_button.png"
}
}
response = requests.post("http://localhost:8000/api/v1/execute", json=action)
Core Conceptsโ
States and Observationsโ
Brobot uses a state-based approach. The MCP server exposes:
- State Structure: The complete map of your application's states
- Observations: Current state with screenshot and confidence scores
Actionsโ
Available actions mirror Brobot's capabilities:
click
: Click on UI elementstype
: Enter textdrag
: Drag and dropwait
: Wait for state changes
Integration Patternsโ
Common patterns for AI integration:
- Autonomous Agent: AI observes and acts independently
- Guided Automation: AI suggests actions for approval
- Hybrid Control: Mix manual and AI-driven steps
Next Stepsโ
- ๐ Read the API Reference for detailed endpoint documentation
- ๐ง Learn about Configuration Options
- ๐ก Explore AI Integration Examples
- ๐ Check Troubleshooting Guide if you encounter issues
Getting Helpโ
- GitHub Issues: Report bugs or request features
- Discord Community: Join our Discord for real-time help
- Documentation: Full docs at brobot.dev