📅 Book a 30-min Demo📞 Call/text (571) 293-0242
Connector · Data and AI platform (lakehouse)

Databricks MCP Server & Agent Skill

Last updated

ibl.ai publishes a free Databricks agent skill: a SKILL.md that teaches an AI agent to read and write Databricks data and expose it as an MCP tool. Download it, connect your own Databricks credentials, and run the agent on ibl.ai Agentic OS, where you own all the code and the data.

Databricks holds an organization's lakehouse data as Unity Catalog catalogs, schemas, tables, volumes and functions, plus SQL warehouses, jobs, notebooks, ML models and vector search indexes.

What is the Databricks API, and what can an agent read from it?

Workspace REST API for jobs, workspace files and Unity Catalog metadata; the SQL Statement Execution API runs SQL on a SQL warehouse and returns results inline or as presigned links.

  • API: Databricks REST API; SQL Statement Execution API; Unity Catalog API (REST)
  • Access: Issued by your administrator
  • Base URL: https://{workspace-hostname}/api/2.0/
  • Authentication: OAuth M2M (service principal); OAuth U2M (user); OAuth token federation; Personal access token

A workspace admin creates a service principal or user, grants CAN USE on a SQL warehouse and Unity Catalog privileges on data, then issues OAuth credentials or a token.

Tokens go in Authorization: Bearer. Databricks recommends OAuth over personal access tokens; token federation avoids managing Databricks secrets. PATs work at workspace level only, not account level.

What can an agent read from Databricks?

  • SQL query results — POST /api/2.0/sql/statements with warehouse_id; INLINE results up to 25 MiB, EXTERNAL_LINKS for larger (JSON, CSV, ARROW_STREAM).
  • Statement status — GET /api/2.0/sql/statements/{statement_id}; wait_timeout defaults to 10s (5–50s).
  • Unity Catalog tables — GET /api/2.1/unity-catalog/tables with catalog_name and schema_name.
  • Natural-language analytics — Genie (system.ai.genie_one_mcp) answers questions over governed data.

What can an agent write to Databricks?

  • DML and DDL — Statements on a SQL warehouse can insert, update, delete or create objects, limited by Unity Catalog grants. Confirm with the user first.
  • Run jobs — Trigger an existing job with /jobs/run-now (20 requests per second).
  • Cancel statement — on_wait_timeout=CANCEL cancels a statement that exceeds wait_timeout.

Limits: Documented per-endpoint limits, e.g. Jobs API run-now 20 req/s and runs/get 100 req/s; DBFS 30 req/s. SQL Statement Execution returns up to 25 MiB inline; use EXTERNAL_LINKS beyond that.

Lakehouse tables often hold customer records, financial data and other regulated data (PII, PHI) depending on what the organization loads; access follows Unity Catalog grants.

How do I connect Databricks to an AI agent over MCP?

Databricks publishes an official MCP server: Databricks managed MCP servers.

Databricks-hosted at https://{workspace}/ai-gateway/mcp-services/{name}: Genie (GA), Databricks SQL (Beta) and others. Legacy /api/2.0/mcp/ endpoints still listed.

You can register that server on ibl.ai as it is, or use the SKILL.md below when you want the agent to call the Databricks REST API; SQL Statement Execution API; Unity Catalog API directly.

On ibl.ai an MCP connection is three records, and a working integration always needs all three:

  1. Register the server — POST …/mcp-servers/ with its URL, a transport (sse, websocket or streamable_http) and an auth_type of none, token or oauth2.
  2. Create a connection that holds the credential, scoped to the whole organization (platform), to a single agent, or to each user (user).
  3. Enable it on the agent — mcp-tool in tool_slugs and the server id in mcp_servers. Both lists are replaced, not merged, so read the settings first.

For an API key or a service credential, one token connection at platform scope serves the whole organization; scope a connection to a single agent when that agent needs narrower access than the rest.

The field reference is in MCP server connections, and the /iblai-api-agent-mcp skill runs all three steps from a coding agent.

What does the Databricks agent skill do?

The Databricks skill is one SKILL.md file named iblai-connect-databricks. It gives an agent the auth conventions, what to read, which writes need a person's confirmation, the ibl.ai MCP wiring, example prompts and guardrails.

Download the Databricks SKILL.md

Save it as .claude/skills/iblai-connect-databricks/SKILL.md, or in your agent's skills folder. The ibl.ai platform skills it calls come from one install:

$ npx skills add iblai/vibe --all

This connector skill is a starting point maintained on ibl.ai, not part of the iblai/vibe repository.

It was checked against Databricks' documentation on October 20, 2026. Test it against your own Databricks setup before you rely on it.

What can I ask a Databricks agent to do?

  • “How many active subscriptions did we have at the end of each month this year?”
  • “List the tables in the main.sales schema and describe what each one holds”
  • “Run the nightly_revenue job now and tell me when it finishes”
  • “Compare churn by plan tier for Q2 versus Q3 from the gold tables”

How do I run a Databricks agent on ibl.ai?

Agentic OS is where the agent runs on ibl.ai: it holds the Databricks MCP connection, the model the agent uses, its instructions and its conversation history.

On ibl.ai you own all the code and the data, run it model-agnostic across any LLM, and pay with no per-seat pricing — so you can deploy anywhere, from your own cloud to a fully air-gapped network.

Which ibl.ai agents work with Databricks data?

Which ibl.ai platform skills does a Databricks agent use?

What else should I know before connecting Databricks to an AI agent?

Is there an official Databricks MCP server?

Yes. Databricks publishes Databricks managed MCP servers.

You can register it on ibl.ai as an MCP server, or use the SKILL.md when you want the agent to call the Databricks REST API; SQL Statement Execution API; Unity Catalog API directly.

Who can get Databricks API credentials?

A workspace admin creates a service principal or user, grants CAN USE on a SQL warehouse and Unity Catalog privileges on data, then issues OAuth credentials or a token.

Does Databricks data leave my infrastructure when an ibl.ai agent uses it?

On ibl.ai, you own all the code and the data. Self-hosted in your cloud or data center, the agent, its MCP connections and its conversation logs stay inside your perimeter.

Because the platform is model-agnostic, you can point the agent at a model you host, so Databricks records need not reach a third-party model provider.

Is the Databricks agent skill free to use?

Yes. The SKILL.md is a free download with no sign-up. It assumes you already have Databricks credentials, and your agreement with Databricks still governs how you use its API.

Which other data sources can an AI agent connect to alongside Databricks?

Which other Enterprise systems have a connector?

Every connector, grouped by sector, is on the connectors hub.