FiftyOne MCP Server

fo_agent

Control FiftyOne datasets through AI assistants using the Model Context Protocol

Overview

Enable ChatGPT and Claude to explore datasets, execute operators, and build computer vision workflows through natural language. This server exposes FiftyOne's operator framework (80+ built-in operators) through 16 MCP tools.

Features

Dataset Management (3 tools) - List, load, and summarize datasets
Operator System (5 tools) - Execute any FiftyOne operator dynamically
Plugin Management (5 tools) - Discover and install FiftyOne plugins
Session Management (3 tools) - Control FiftyOne App for delegated execution
Natural Language Workflows - Multi-step operations through conversation
ChatGPT & Claude Compatible - Works with desktop apps

Quick Start

Option 1: pip (Simplest)

pip install fiftyone-mcp-server

Then add to your AI tool config and restart:

Claude Desktop

Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "fiftyone": {
      "command": "fiftyone-mcp"
    }
  }
}

Claude Code

claude mcp add fiftyone -- fiftyone-mcp

Cursor

Add to Cursor MCP settings:

{
  "fiftyone": {
    "command": "fiftyone-mcp"
  }
}

VSCode

Add to .vscode/mcp.json:

{
  "servers": {
    "fiftyone": {
      "command": "fiftyone-mcp"
    }
  }
}

ChatGPT Desktop

Edit ~/Library/Application Support/ChatGPT/config.json:

{
  "mcpServers": {
    "fiftyone": {
      "command": "fiftyone-mcp"
    }
  }
}

Option 2: uvx (No Install Needed)

If you have uv installed:

{
  "mcpServers": {
    "fiftyone": {
      "command": "uvx",
      "args": ["fiftyone-mcp-server"]
    }
  }
}

This downloads and runs the latest version automatically.

Usage

After configuration, restart your AI assistant and try:

"List all my datasets"
"Load quickstart dataset and show summary"
"What operators are available for managing samples?"
"Set context to my dataset, then tag high-confidence samples"
"What plugins are available? Install the brain plugin"
"Find similar images in my dataset"

The server starts with 50 built-in operators. Install plugins to expand functionality - the AI can discover and install plugins automatically when needed (brain, zoo, annotation, evaluation, and more).

Architecture

Component	Description
Operator System	80+ FiftyOne operators through unified interface
Plugin System	AI discovers and installs plugins on demand
Session System	Launch FiftyOne App for delegated operators
Context Management	Dataset, view, and selection state

Design Philosophy: Minimal tool count (16 tools), maximum flexibility (full operator & plugin ecosystem).

Contributing

We welcome contributions! Here's how to set up a local development environment.

Local Development Setup

# Clone the repository
git clone https://github.com/AdonaiVera/fiftyone-mcp-server.git
cd fiftyone-mcp-server

# Install Poetry (if not installed)
curl -sSL https://install.python-poetry.org | python3 -

# Install dependencies
poetry install

# Run the server locally
poetry run fiftyone-mcp

Testing Your Changes

# Run tests
poetry run pytest

# Code formatting
poetry run black -l 79 src/

# Linting
poetry run pylint --errors-only src/

# Test with MCP Inspector
npx @modelcontextprotocol/inspector poetry run fiftyone-mcp

Using Local Version with Claude

To test your local changes with Claude Desktop, update your config:

{
  "mcpServers": {
    "fiftyone": {
      "command": "poetry",
      "args": ["run", "fiftyone-mcp"],
      "cwd": "/absolute/path/to/fiftyone-mcp-server"
    }
  }
}

Resources

Built with FiftyOne and Model Context Protocol

fiftyone-mcp-server

Documentation