---
name: iblai-connect-databricks
description: "Read and write Databricks data through the Databricks REST API; SQL Statement Execution API; Unity Catalog API and expose it to an ibl.ai agent as an MCP tool. Use when the user asks an agent to work with Databricks data, or to set up a Databricks MCP connection on ibl.ai."
metadata:
  kind: guide
---

# Databricks connector for AI agents

> A starting point maintained on ibl.ai at https://ibl.ai/developer/connectors/databricks — not part of the iblai/vibe repository. Checked against Databricks' documentation on October 20, 2026; test it against your own Databricks setup before you rely on it.

Databricks holds an organization's lakehouse data as Unity Catalog catalogs, schemas, tables, volumes and functions, plus SQL warehouses, jobs, notebooks, ML models and vector search indexes.

## Auth & conventions

- **API:** Databricks REST API; SQL Statement Execution API; Unity Catalog API (REST). Workspace REST API for jobs, workspace files and Unity Catalog metadata; the SQL Statement Execution API runs SQL on a SQL warehouse and returns results inline or as presigned links.
- **Base URL:** https://{workspace-hostname}/api/2.0/
- **Access:** Issued by your administrator. A workspace admin creates a service principal or user, grants CAN USE on a SQL warehouse and Unity Catalog privileges on data, then issues OAuth credentials or a token.
- **Auth:** OAuth M2M (service principal); OAuth U2M (user); OAuth token federation; Personal access token. Tokens go in Authorization: Bearer. Databricks recommends OAuth over personal access tokens; token federation avoids managing Databricks secrets. PATs work at workspace level only, not account level.
- **Limits:** Documented per-endpoint limits, e.g. Jobs API run-now 20 req/s and runs/get 100 req/s; DBFS 30 req/s. SQL Statement Execution returns up to 25 MiB inline; use EXTERNAL_LINKS beyond that.
- Keep Databricks credentials in environment variables or the ibl.ai credential store. Never print a token in chat and never commit one to a file.
- Destructive or outward-facing calls are marked **Confirm with the user first** — do.

## Reads

- **SQL query results** — POST /api/2.0/sql/statements with warehouse_id; INLINE results up to 25 MiB, EXTERNAL_LINKS for larger (JSON, CSV, ARROW_STREAM).
- **Statement status** — GET /api/2.0/sql/statements/{statement_id}; wait_timeout defaults to 10s (5–50s).
- **Unity Catalog tables** — GET /api/2.1/unity-catalog/tables with catalog_name and schema_name.
- **Natural-language analytics** — Genie (system.ai.genie_one_mcp) answers questions over governed data.

## Writes

- **DML and DDL** — Statements on a SQL warehouse can insert, update, delete or create objects, limited by Unity Catalog grants. **Confirm with the user first.**
- **Run jobs** — Trigger an existing job with /jobs/run-now (20 requests per second).
- **Cancel statement** — on_wait_timeout=CANCEL cancels a statement that exceeds wait_timeout.

## Expose it to an ibl.ai agent (MCP)

Databricks publishes an official MCP server, Databricks managed MCP servers (https://docs.databricks.com/aws/en/generative-ai/mcp/managed-mcp). Register it below as-is, or host your own server that wraps the calls in this skill.

First install the ibl.ai platform skills and connect your organization (writes `IBLAI_ORG`, `IBLAI_USERNAME` and `IBLAI_API_KEY` to `.env`):

```bash
npx skills add iblai/vibe --all
# then, in your coding agent:
/iblai-api-login
```

Every call below sends `Authorization: Api-Token $IBLAI_API_KEY`.

1. **Register the server** — `POST https://api.iblai.app/dm/api/ai-mentor/orgs/$IBLAI_ORG/users/$IBLAI_USERNAME/mcp-servers/` with `name`, `url`, `transport` (`sse` | `websocket` | `streamable_http`) and `auth_type` (`none` | `token` | `oauth2`). Capture the returned `id`.
   - For an API key or service credential use `auth_type=token` with `auth_scope=platform` for one key across the organization, or scope it to a single agent for one key per agent.
2. **Create the connection** — `POST https://api.iblai.app/dm/api/ai-mentor/orgs/$IBLAI_ORG/users/$IBLAI_USERNAME/mcp-server-connections/` with `server`, `scope` (`platform` | `mentor` | `user`), `auth_type`, and either `credentials` + `authorization_scheme` or a `connected_service` id for OAuth. Credentials are masked on read; never send a masked value back.
3. **Enable it on the agent** — `GET https://api.iblai.app/dm/api/ai-mentor/orgs/$IBLAI_ORG/users/$IBLAI_USERNAME/mentors/{agent_id}/settings/`, add `mcp-tool` to `tool_slugs` and the server id to `mcp_servers`, then `PATCH` the full lists back. They are replaced, not merged.

Full reference: https://ibl.ai/developer/platform/mcp/mcp-connections and the `/iblai-api-agent-mcp` skill.

## Example prompts

- "How many active subscriptions did we have at the end of each month this year?"
- "List the tables in the main.sales schema and describe what each one holds"
- "Run the nightly_revenue job now and tell me when it finishes"
- "Compare churn by plan tier for Q2 versus Q3 from the gold tables"

## Guardrails

- Lakehouse tables often hold customer records, financial data and other regulated data (PII, PHI) depending on what the organization loads; access follows Unity Catalog grants.
- Request the narrowest Databricks scopes or role the task needs, and prefer a read-only credential for agents that only answer questions.
- Return the fields the user asked for, not whole records; do not paste personal data into chat that the user did not request.
- Before any write, show the user exactly what will change and wait for a yes.
- Respect the documented limits above: page through results and back off on rate-limit responses instead of retrying in a loop.

## References

- Databricks authentication: https://docs.databricks.com/aws/en/dev-tools/auth/
- SQL Statement Execution API tutorial: https://docs.databricks.com/aws/en/dev-tools/sql-execution-tutorial
- Managed MCP servers: https://docs.databricks.com/aws/en/generative-ai/mcp/managed-mcp
- Databricks managed MCP servers: https://docs.databricks.com/aws/en/generative-ai/mcp/managed-mcp
- This connector on ibl.ai: https://ibl.ai/developer/connectors/databricks
- ibl.ai MCP server connections: https://ibl.ai/developer/platform/mcp/mcp-connections