Databricks Managed MCP Servers Guide: Developer Setup and Unity Catalog Integration

Q: Can multiple AI clients connect to the same MCP server simultaneously?

Yes. The databricks-mcp serve command starts a local proxy process that handles multiple simultaneous connections. The underlying Databricks managed server is serverless and scales horizontally — there's no single-server bottleneck. Each client connection maintains its own authenticated session with its own OAuth token, so concurrent users see their own permission-filtered results.

Databricks managed MCP servers give AI agents secure, governed access to your Lakehouse data — Genie (NL-to-SQL), Vector Search, and UC Functions — with zero infrastructure overhead and Unity Catalog permissions enforced automatically on every call.

What Are Databricks Managed MCP Servers?

Databricks managed MCP servers are hosted, serverless endpoints that expose Lakehouse capabilities — structured data queries, vector search, and custom functions — to any MCP-compatible AI client through the Model Context Protocol standard. Unlike self-hosted MCP servers that require you to provision infrastructure, manage TLS, and handle scaling, Databricks-managed servers run entirely on Databricks serverless compute with on-behalf-of-user authentication baked in. Every tool call automatically inherits the caller’s Unity Catalog permissions, which means a data analyst connecting Claude Desktop to a Genie space can only query tables their UC role allows — no manual ACL syncing required. Databricks announced general availability of managed MCP servers in early 2026 alongside a broader “Week of Agents” initiative, and the platform has seen multi-agent workflow usage grow 327% in four months. The practical upshot for developers: you get enterprise-grade governance without writing a single line of server-side authentication code.

Three Types of Databricks MCP Servers: Managed, External, and Custom

Databricks supports three MCP deployment models, and choosing the right one shapes your architecture from day one. Managed servers (Genie, Vector Search, UC Functions) are hosted by Databricks, require no setup, and auto-enforce Unity Catalog permissions. You simply point your client at the server URL. External servers are third-party MCP servers accessible via the MCP Catalog and Databricks Marketplace — partners like You.com, Glean, S&P Global, Factset, Dun & Bradstreet, and Moody’s have published tools you can connect with single-click configuration. Custom servers are MCP servers you build and host on Databricks Apps using the databricks-mcp Python package; these give you full control over tool definitions but require deployment and maintenance. For most enterprise developers, managed servers cover 80% of use cases. Custom servers make sense when you need proprietary business logic exposed as MCP tools. External marketplace servers extend your agent’s reach to financial data, search, and external knowledge bases.

Type	Setup effort	Governance	Best for
Managed (Genie/VS/UC)	Zero — just configure client	Automatic UC enforcement	Lakehouse data access
External (Marketplace)	Single-click config	Partner-defined	Third-party data/tools
Custom (Databricks Apps)	Python dev + deployment	Developer-controlled	Proprietary logic

Prerequisites and Environment Setup (Python 3.12+, Dependencies)

Getting your local environment ready for Databricks MCP development requires Python 3.10 at minimum, though 3.12+ is strongly recommended because the mcp package leverages newer async primitives. You’ll also need serverless compute enabled in your Databricks workspace — without it, managed MCP endpoints won’t activate. The core dependency set is small but version-pinned: mcp>=1.9 for the MCP protocol implementation, databricks-sdk[openai] for workspace client and auth helpers, mlflow>=3.1.0 for experiment tracking integration, databricks-agents>=1.0.0 for the agent framework, and databricks-mcp for the CLI and Python client. These pinned minimums exist because Databricks MCP uses the Streamable HTTP transport introduced in MCP 1.9 — older clients that only support SSE transport won’t connect. Before writing any code, verify your workspace URL and that your user account has the USE CATALOG and USE SCHEMA Unity Catalog privileges on the catalogs you want to expose.

# Create isolated environment
python3.12 -m venv .venv && source .venv/bin/activate

# Install all required packages
pip install "mcp>=1.9" "databricks-sdk[openai]" "mlflow>=3.1.0" \
  "databricks-agents>=1.0.0" databricks-mcp

# Verify installation
python -c "import databricks_mcp; print(databricks_mcp.__version__)"

Verifying Workspace Serverless Compute

Before configuring any MCP server, check that serverless compute is on. In the Databricks workspace UI, go to Settings > Compute > Serverless and confirm the toggle is enabled. Alternatively, use the SDK: WorkspaceClient().settings.default_namespace_setting.get() — if it throws a permission error rather than returning a result, serverless may be disabled at the workspace level and you’ll need an admin to enable it.

Authenticating to Your Databricks Workspace (OAuth and PAT)

Databricks MCP supports two authentication paths for local development: OAuth (recommended) and Personal Access Tokens (PAT). OAuth uses the Databricks CLI’s browser-based login flow and issues short-lived tokens that automatically refresh — this is the correct choice for human-in-the-loop developer workflows because it ties every MCP tool call to a specific user identity, which Unity Catalog can then audit. PAT-based auth uses a static token stored in environment variables; it’s simpler to configure but creates a shared service account identity that bypasses per-user audit trails. For production agent deployments where no human is in the loop, service principal OAuth (M2M) is the right answer — not PATs. The databricks-mcp CLI detects which auth method to use based on environment variables, following the same credential chain as the Databricks SDK: environment variables → ~/.databrickscfg → Azure Managed Identity (on Azure) → instance profile (on AWS).

# OAuth login (recommended for local dev)
databricks auth login --host https://your-workspace.azuredatabricks.net

# Verify auth works
databricks auth env --host https://your-workspace.azuredatabricks.net

# PAT alternative (less preferred)
export DATABRICKS_HOST=https://your-workspace.azuredatabricks.net
export DATABRICKS_TOKEN=dapi...your-token-here

Service Principal OAuth for Production

For unattended agent deployments, use M2M OAuth with a service principal. Create a service principal in your workspace, assign Unity Catalog roles to it, and configure OAuth credentials via the Databricks admin console. Set DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET environment variables instead of DATABRICKS_TOKEN. The SDK automatically detects M2M credentials and uses the client credentials flow, keeping tokens rotated without manual intervention.

Installing databricks-mcp and Configuring Your First Server

The databricks-mcp package ships a CLI that generates MCP server configuration for all three managed server types. After installation, use databricks-mcp serve to start a local proxy that translates MCP protocol calls into Databricks API requests — this is what you configure external clients like Claude Desktop to point at. The configuration file (mcp.json in VS Code or the JSON config in Claude Desktop) specifies the command to run, the workspace host, and any server-specific parameters like the Genie space ID or vector search endpoint name. Generating the config requires knowing your workspace URL and, for Genie, the space ID visible in the Databricks UI under Genie > Spaces.

# Generate MCP config for all managed servers
databricks-mcp config generate \
  --host https://your-workspace.azuredatabricks.net \
  --output ./mcp-config.json

# Start a local MCP proxy (foreground, useful for debugging)
databricks-mcp serve \
  --host https://your-workspace.azuredatabricks.net \
  --server genie \
  --genie-space-id your-space-id

The generated mcp-config.json follows the standard MCP client configuration format and can be dropped directly into VS Code’s .vscode/mcp.json or Claude Desktop’s config file without modification.

Working with Genie Spaces (Natural Language to SQL)

Genie is Databricks’ natural-language-to-SQL service, and its MCP integration exposes a single high-value tool: genie_execute_message. You send a plain-English question — “What were the top 10 revenue-generating products in Q1 2026?” — and Genie translates it to SQL, executes it against tables in your Unity Catalog, and returns structured results. The critical architectural point is that Genie still enforces the caller’s row-level security filters and column masks defined in Unity Catalog — a user without access to the orders.pii_columns column cannot retrieve that data through Genie, even if they phrase the question cleverly. Genie spaces are pre-configured by data teams with approved tables, semantic definitions, and query examples; developers connect to an existing space rather than defining schema from scratch. To find your space ID, navigate to Genie > Spaces in the workspace UI — the ID appears in the URL as a UUID. Setting up Genie MCP takes under five minutes once your space exists.

# Direct SDK usage (useful for testing before configuring MCP client)
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
space_id = "your-genie-space-id"

# Start a conversation
conversation = w.genie.start_conversation(space_id=space_id)

# Ask a question
message = w.genie.create_conversation_message(
    space_id=space_id,
    conversation_id=conversation.conversation_id,
    content="What were the top 5 products by revenue last month?",
)
print(message.attachments)

Genie Limitations and Best Practices

Genie performs best on well-defined analytical questions against structured tables. It struggles with multi-hop joins across more than four tables, ambiguous column names, and questions requiring real-time streaming data. Always configure your Genie space with SQL examples (at least 10-20 representative queries) to ground the model’s SQL generation. When a Genie question returns unexpected results, check the space’s Query History tab — Genie logs every generated SQL statement, making debugging straightforward.

Vector Search Integration for Unstructured Data Retrieval

Databricks Vector Search MCP exposes semantic similarity search over Delta Lake tables containing embeddings — the practical use case is retrieval-augmented generation (RAG) where an AI agent needs to find relevant documents, code snippets, or knowledge base articles before answering a question. The Vector Search MCP tool (vector_search_query) takes a natural-language query, converts it to an embedding using the endpoint you’ve configured in your workspace, and returns the top-k most similar chunks with their metadata and similarity scores. Like Genie, Vector Search respects Unity Catalog permissions: the calling user must have SELECT on the source Delta table that backs the vector index. Vector Search indexes can be either Delta Sync (auto-synced from a Delta table as data changes) or Direct Vector Access (you manage embedding updates via API). For most RAG use cases, Delta Sync indexes are the right choice — new documents added to the source table are automatically embedded and indexed within minutes.

# Test vector search before wiring up MCP
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

results = w.vector_search_indexes.query_index(
    index_name="main.rag_catalog.docs_index",
    columns=["content", "source_url", "last_updated"],
    query_text="databricks unity catalog row level security",
    num_results=5,
)

for result in results.result.data_array:
    print(result)

Combining Genie and Vector Search in a Single Agent

The most powerful pattern is routing: a supervisor agent receives a user question, classifies it as structured (SQL/Genie) or unstructured (RAG/Vector Search), and dispatches to the appropriate MCP tool. Databricks’ Supervisor Agent became the #1 Agent Bricks use case within 4 months of launch (capturing 37% of all Agent Bricks usage), largely because this routing pattern maps cleanly onto Databricks’ tool set.

Unity Catalog Functions (UC Functions) as MCP Tools

Unity Catalog Functions are Python or SQL functions registered in the UC metastore that can be discovered, governed, and executed like any other UC asset. The UC Functions MCP server (uc_functions) automatically converts every function in your specified catalog/schema into an MCP tool, with the function signature becoming the tool’s input schema and the UC comment becoming the tool description. This means a Python function like calculate_churn_risk(customer_id: str) -> float registered at main.ml_tools.calculate_churn_risk becomes a callable MCP tool with a properly typed input schema — no additional wrapping required. Unity Catalog handles permission checks on each invocation: the calling user needs EXECUTE privilege on the function. Organizations with unified AI governance ship 12x more AI projects than those without, and UC Functions as MCP tools is a direct mechanism for that governance — every tool invocation is logged in UC’s audit trail.

-- Register a UC Function (SQL example)
CREATE OR REPLACE FUNCTION main.ml_tools.get_customer_tier(
  customer_id STRING COMMENT "Unique customer identifier"
)
RETURNS STRING
COMMENT "Returns customer tier (Bronze/Silver/Gold/Platinum) based on LTV"
LANGUAGE SQL
AS $$
  SELECT
    CASE
      WHEN lifetime_value >= 50000 THEN 'Platinum'
      WHEN lifetime_value >= 10000 THEN 'Gold'
      WHEN lifetime_value >= 1000  THEN 'Silver'
      ELSE 'Bronze'
    END
  FROM main.customers.profiles
  WHERE id = customer_id
$$;

# Register a Python UC Function
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import FunctionInfo

# After defining your function in a notebook or via SDK,
# it appears automatically in the UC Functions MCP server
# — no extra registration step needed

Unity Catalog Governance and Permission Enforcement

Unity Catalog’s permission model flows transparently through every Databricks MCP tool call. When a user invokes an MCP tool, the Databricks MCP server issues the underlying API call on behalf of that user using the OAuth token from the initial authentication. Unity Catalog evaluates the caller’s privileges — SELECT for Genie and Vector Search queries, EXECUTE for UC Functions — and applies row-level security filters, column masks, and data lineage tracking exactly as it would for a direct SQL query. This on-behalf-of-user design is the key differentiator from service-account-based MCP approaches: you get full audit trails showing which user queried which data, when, and through which AI client. For organizations in regulated industries (financial services, healthcare), this is often a hard compliance requirement. The Databricks audit log records MCP tool calls under the databricks_mcp action category, making it straightforward to build SIEM dashboards that monitor AI agent data access alongside traditional BI tool access.

Setting Up Unity Catalog Permissions for MCP

-- Grant Genie space access (table-level)
GRANT SELECT ON TABLE main.sales.orders TO `data-analyst-group`;

-- Grant Vector Search index access
GRANT SELECT ON TABLE main.rag_catalog.docs_source TO `developer-group`;

-- Grant UC Function execution
GRANT EXECUTE ON FUNCTION main.ml_tools.calculate_churn_risk TO `ml-engineer-group`;

-- Optional: restrict specific columns via column mask
ALTER TABLE main.sales.orders
ALTER COLUMN customer_email
SET MASK main.masks.email_mask
  USING COLUMNS (customer_email)
  FOR (IS_MEMBER('data-analyst-group'));

Connecting External Clients: Claude Desktop, VS Code, GitHub Copilot

Any MCP-compatible client can connect to Databricks managed MCP servers in under 10 minutes. The configuration pattern is the same across clients: specify the databricks-mcp CLI command as the server process, and pass your workspace host and server type as arguments. Claude Desktop stores MCP config in ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows). VS Code reads from .vscode/mcp.json in the workspace root, and GitHub Copilot uses a similar JSON format in the Copilot settings file.

// Claude Desktop: ~/Library/Application Support/Claude/claude_desktop_config.json
{
  "mcpServers": {
    "databricks-genie": {
      "command": "databricks-mcp",
      "args": [
        "serve",
        "--host", "https://your-workspace.azuredatabricks.net",
        "--server", "genie",
        "--genie-space-id", "your-space-uuid"
      ]
    },
    "databricks-vector-search": {
      "command": "databricks-mcp",
      "args": [
        "serve",
        "--host", "https://your-workspace.azuredatabricks.net",
        "--server", "vector_search",
        "--index-name", "main.rag_catalog.docs_index"
      ]
    },
    "databricks-uc-functions": {
      "command": "databricks-mcp",
      "args": [
        "serve",
        "--host", "https://your-workspace.azuredatabricks.net",
        "--server", "uc_functions",
        "--catalog", "main",
        "--schema", "ml_tools"
      ]
    }
  }
}

// VS Code: .vscode/mcp.json
{
  "servers": {
    "databricks": {
      "type": "stdio",
      "command": "databricks-mcp",
      "args": ["serve", "--host", "${env:DATABRICKS_HOST}", "--server", "genie",
               "--genie-space-id", "${env:GENIE_SPACE_ID}"]
    }
  }
}

Troubleshooting Client Connections

The most common connection failure is expired OAuth tokens — run databricks auth login again if you see 401 Unauthorized in the MCP server logs. The second most common issue is missing serverless compute: check the Databricks workspace admin settings if tools connect but every call returns a ComputeNotAvailable error.

Using the MCP Catalog and Databricks Marketplace

The Databricks MCP Catalog (Beta) is a centralized registry for discovering and governing MCP servers within your organization — think of it as an internal App Store for AI tools. Teams register their custom MCP servers (built on Databricks Apps or elsewhere) in the catalog with descriptions, input schemas, and access policies. Other teams can then discover and connect to these tools without needing to know deployment details. The Databricks Marketplace extends this further: external partners have published MCP servers for financial data (S&P Global, Factset, Moody’s, Dun & Bradstreet), enterprise search (Glean), and web search (You.com). Connecting a marketplace MCP server uses single-click configuration in the workspace UI — Databricks handles credential exchange and connection management. This is the fastest path to giving your AI agents access to high-value external data sources without building custom connectors.

Registering a Custom Server in MCP Catalog

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Register your custom MCP server in the catalog
w.mcp_catalog.register_server(
    name="my-team-crm-tools",
    description="CRM data access tools for sales team agents",
    server_url="https://your-databricks-app.azurewebsites.net/mcp",
    auth_type="unity_catalog",
    tags=["crm", "sales", "production"],
)

Monitoring MCP Usage with AI Gateway

Databricks AI Gateway provides a unified control plane for monitoring, rate limiting, and access management across all MCP tool calls in your workspace. When you route MCP traffic through AI Gateway, every tool invocation generates a structured log entry containing: calling user, tool name, input parameters (redacted for PII if configured), response latency, token usage (for LLM-backed tools like Genie), and Unity Catalog audit metadata. AI Gateway also enforces rate limits per user or team — preventing a runaway agent from monopolizing Genie space compute. Setting up AI Gateway for MCP requires creating an AI Gateway endpoint in the workspace UI and pointing your MCP server configuration at the gateway URL instead of the direct API endpoint. The latency overhead is typically under 10ms for the gateway layer, negligible compared to Genie query execution time (usually 2-15 seconds).

# Query AI Gateway usage logs for MCP calls
from databricks.sdk import WorkspaceClient
import pandas as pd

w = WorkspaceClient()

# AI Gateway logs are surfaced in system tables
df = spark.sql("""
  SELECT
    timestamp,
    request_metadata.user_id,
    request_metadata.tool_name,
    response_metadata.latency_ms,
    response_metadata.status_code
  FROM system.ai_gateway.mcp_request_logs
  WHERE timestamp >= current_date() - INTERVAL 7 DAYS
  ORDER BY timestamp DESC
  LIMIT 1000
""")

display(df)

Building Custom MCP Servers on Databricks Apps

When managed servers don’t cover your use case — proprietary ML model inference, custom ETL triggers, or business logic not expressible as UC Functions — you can build and host MCP servers on Databricks Apps. The key constraint is transport: Databricks Apps uses HTTP, so your MCP server must use Streamable HTTP transport (introduced in MCP 1.9) rather than stdio. The databricks-mcp package provides base classes that handle authentication, Unity Catalog permission checks, and the Streamable HTTP transport layer; you define only the tool implementations. Apps automatically scale with traffic and integrate with workspace IAM for authentication — users authenticate via OAuth the same way they would with managed servers.

# app.py — minimal Databricks Apps MCP server
from databricks_mcp import DatabricksMCPServer, tool
from databricks.sdk import WorkspaceClient

app = DatabricksMCPServer(name="my-custom-tools", version="1.0.0")
w = WorkspaceClient()

@tool(
    name="get_pipeline_status",
    description="Returns the current status and last run time for a Delta Live Tables pipeline",
)
def get_pipeline_status(pipeline_id: str) -> dict:
    pipeline = w.pipelines.get(pipeline_id=pipeline_id)
    return {
        "name": pipeline.name,
        "state": pipeline.state.value,
        "last_modified": str(pipeline.last_modified),
    }

# Databricks Apps expects a WSGI/ASGI app bound to PORT env var
if __name__ == "__main__":
    import uvicorn, os
    uvicorn.run(app.asgi_app, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))

# databricks.yml — deployment config for Databricks Apps
bundle:
  name: custom-mcp-server
  
resources:
  apps:
    custom-mcp:
      name: custom-mcp-server
      source_code_path: ./app
      description: Custom MCP server for pipeline management tools

Performance Considerations for Custom Servers

Cold-start latency on Databricks Apps ranges from 1-3 seconds when the app hasn’t received traffic in several minutes. For MCP tool calls in interactive AI workflows, this cold-start is noticeable. Mitigate it with a lightweight health-check endpoint that your monitoring system pings every 30 seconds to keep the app warm, or use the Databricks Apps “always-on” option (available at higher pricing tiers).

Best Practices for Enterprise MCP Deployments

Enterprise-scale MCP deployments on Databricks require intentional decisions around governance, performance, and observability from day one — retrofitting these later is significantly more expensive. The governance foundation is straightforward: always use OAuth rather than PATs, never share service principal credentials across teams, and register every MCP tool in the MCP Catalog so there’s a discoverable inventory. Only 19% of organizations have deployed AI agents in production despite heavy interest, and the gap is often governance: teams that treat MCP tools as governed UC assets from the start close that gap faster. For performance, configure connection pooling in the databricks-mcp server (the --max-connections flag) to prevent per-tool-call connection overhead. For observability, route all MCP traffic through AI Gateway even if you’re not using rate limiting — the audit logs alone justify the minimal latency cost. Finally, test Unity Catalog permission boundaries before your first production deployment: impersonate different user roles in a staging workspace and verify that UC masks and row filters behave as expected through the MCP interface.

Checklist for Production Readiness

OAuth (not PAT) auth configured for all client connections
Service principals have minimum required UC privileges
AI Gateway endpoint configured and MCP traffic routed through it
Rate limits set per team or user group in AI Gateway
All custom MCP servers registered in the MCP Catalog
Unity Catalog row-level security tested with impersonation
Databricks Apps health-check endpoint configured to prevent cold starts
SIEM integration for databricks_mcp audit log category
Genie space query examples updated as data schemas evolve
Vector Search index sync lag monitored (alert if > 15 minutes)

FAQ

Can I use Databricks managed MCP servers without Unity Catalog?

No. Unity Catalog must be enabled in your workspace for managed MCP servers to function — it provides the permission enforcement layer that every tool call relies on. If your workspace uses the legacy Hive metastore, you’ll need to migrate to Unity Catalog before using managed MCP servers.

What’s the difference between Databricks managed MCP and building your own MCP server from scratch?

Managed MCP servers are zero-setup, hosted by Databricks, and automatically enforce Unity Catalog governance. Building your own MCP server (on Databricks Apps or elsewhere) gives you full control over tool definitions and business logic but requires deployment, maintenance, and explicit permission checks. Start with managed servers; build custom ones only when you have requirements managed servers can’t meet.

How does on-behalf-of-user authentication work in practice?

When a user connects Claude Desktop to a Databricks managed MCP server, they authenticate via the Databricks OAuth flow in their browser. The resulting short-lived OAuth token is stored locally by the databricks-mcp CLI. Every subsequent tool call sends this token to the Databricks API, which validates it and executes the request as that specific user — not as a shared service account. Unity Catalog then evaluates that user’s specific permissions for the requested resource.

Can multiple AI clients connect to the same MCP server simultaneously?

Yes. The databricks-mcp serve command starts a local proxy process that handles multiple simultaneous connections. The underlying Databricks managed server is serverless and scales horizontally — there’s no single-server bottleneck. Each client connection maintains its own authenticated session with its own OAuth token, so concurrent users see their own permission-filtered results.

Is Databricks MCP available on AWS, Azure, and GCP?

Managed MCP servers are available on all three cloud providers where Databricks operates, as long as serverless compute is enabled in your workspace. The Python package, CLI, and configuration format are identical across clouds — only the workspace host URL differs. AI Gateway and MCP Catalog (Beta) availability may lag slightly across clouds; check the Databricks release notes for per-cloud GA dates.

What Are Databricks Managed MCP Servers?#

Three Types of Databricks MCP Servers: Managed, External, and Custom#

Prerequisites and Environment Setup (Python 3.12+, Dependencies)#

Verifying Workspace Serverless Compute#

Authenticating to Your Databricks Workspace (OAuth and PAT)#

Service Principal OAuth for Production#

Installing databricks-mcp and Configuring Your First Server#

Working with Genie Spaces (Natural Language to SQL)#

Genie Limitations and Best Practices#

Vector Search Integration for Unstructured Data Retrieval#

Combining Genie and Vector Search in a Single Agent#

Unity Catalog Functions (UC Functions) as MCP Tools#

Unity Catalog Governance and Permission Enforcement#

Setting Up Unity Catalog Permissions for MCP#

Connecting External Clients: Claude Desktop, VS Code, GitHub Copilot#

Troubleshooting Client Connections#

Using the MCP Catalog and Databricks Marketplace#

Registering a Custom Server in MCP Catalog#

Monitoring MCP Usage with AI Gateway#

Building Custom MCP Servers on Databricks Apps#

Performance Considerations for Custom Servers#

Best Practices for Enterprise MCP Deployments#

Checklist for Production Readiness#

FAQ#