Last updated

Where every piece of data comes from

Mirrored from iblai/ontology · docs/data-sources.md. This page is generated — edit it in the repository, not here.

Scope: everything dynamic the console renders, and where each thing it writes is saved.

The rule

This Next.js app has no database. Like the OS app, it keeps nothing of its own beyond the files the server runtime reads. Every value on screen comes from one of three places, and nothing else:

TagSystemWhere it livesHow the browser reaches it
IBLibl.ai platform APIibl.aiDirectly from the browser with the SSO session (axd_token / dm_token from login.ibl.ai), exactly as os.ibl.ai does.
AONVIDIA auto-ontologyOne per platform, own Postgres + pgvector; host's own /api/* on 127.0.0.1:3100Through this app's route handlers, which add the engine's service key (x-api-key). Ports 3001 and 3002 are never called.
TBthe host and MCP ToolboxFiles under $ONTOLOGY_HOME/platforms/<platform>/; one Toolbox process per serverThrough this app's route handlers.

The host is the platform's own server, the same process that serves the pages (install.md). A console hosted elsewhere (the pages-only build) serves pages only and forwards every route handler call to the platform's server (network.md).

If something was not available from one of these, it was removed from the UI or shows “—”. The removals are listed at the end.

The engine gives no row counts, no primary keys, no ingest status and no log over HTTP. Rows show “—”. A key column is the one other tables' foreign keys point at. A system is “ingesting” until its tables appear.

Where each thing is saved

ThingSaved inNotes
Connected databasesthe engine's own Postgres (the postgres service of its compose stack, database auto_ontology)The whole connection, password included, as unencrypted JSON on the catalog row. Set the engine's four VAULT_* variables to keep credentials in Vault instead.
The same database password, for servingenv on the host, mode 0600Toolbox needs it to run tools, and the engine never gives a password back.
Custom HTTP APIsapis.yaml; the key in envA file, as Toolbox reads it. The engine does not take APIs.
Terms, tables, joins, tags, certificationthe engine's Postgres + pgvectorIts semantic layer.
MCP servers (config, flags, revisions)servers/<slug>/Files. The list of running processes is the pid files.
Tool calls and refusals the gateway sawservers/<slug>/gateway.logA log beside Toolbox's own toolbox.log. The Runtime Log, the call counters and the Activity table read it.
Roles and policies, API keys, the server registered for an agentibl.aiWritten from the browser with the admin's own session.

The files are in the platform's own folder, $ONTOLOGY_HOME/platforms/<platform>/, and the engine is the platform's own (network.md). ONTOLOGY_HOME defaults to .ontology in the console's directory.

Screen by screen

Shell

ElementSourceDetail
Logo, header, sidebarstaticibl.ai brand assets.
Platform menuIBLThe platforms the user belongs to.
User / Admin switch, avatarIBLSession and RBAC (admin mode as in the OS).
Notifications, Invites, Management, API, Secrets, Advanced, SupportIBLThe OS's own panels.
Sidebar countsAO / TB / IBLSystems = engine connections; catalog = tables the engine read; ontology = engine terms; MCP gateway = tools across the servers' tools.yaml; agents = ibl.ai agents with a server attached.

Dashboard

ElementSource
Systems connectedAO GET /api/connections
Object types certifiedAO GET /api/terms (name_certified, description_certified)
Tools publishedTB tools across the servers' tools.yaml
Agent tool calls · 24hTB each server's gateway.log
ActivityTB deploys (server revisions); AO compiles and certifications; IBL agent connections
MCP server cardsTB servers/<slug>/ (revision, toolsets, tools, process status); IBL agents attached to the server

Systems (step 1)

ElementSource
Connected databasesAO GET /api/connections. The name is the database's name; the URL never shows the password.
Table and PII countsAO GET /api/exploration/graph and the column tags the PII detector sets
Ingest statusDerived: “ingesting” until the system's tables appear, “compiling” while AO GET /api/semantic-compilation/status says a compile runs
HTTP API systemsTB apis.yaml (served only, never dissected)
Catalog of kindsstatic: what the engine dissects and what Toolbox serves
Test ConnectionAO POST /api/connections/test
Connect & DissectThe password to TB env; AO PUT /api/configurations/pii-detection and …/distinct-value-probing; AO POST /api/connections, which starts the ingest
Fetch SpecThe host fetches the OpenAPI document under the same address rule as the API's base address (the host's own machine outside production only), with the auth header when the document is on the API's own address, and lists its operations as endpoints
Add & ExposeTB apis.yaml, the key in env
PrebuiltOne raw toolset per connected database (<system>-raw); the ticked ones open the server builder already ticked
Database credentialsEntered once. Stored by the engine (its connection row, or Vault) and in TB env for serving. Never in the browser after submit, never on ibl.ai.

Map (step 2)

ElementSource
Pipeline stagesDerived from what the engine has read: tables, columns, terms, links, compile finished (AO GET /api/semantic-compilation/status), tools published (TB server configs)
LogThe changes the console observed in those readings while the page was open. The engine streams no log.
Tables, foreign keys, system clustersAO GET /api/exploration/graph
Terms, term ↔ table mapping, term linksAO GET /api/exploration/semantic-graph (a link is two terms sharing a table)
Table detailAO GET /api/columns/{table_id}, GET /api/exploration/tables/{table_id}/details. Rows show “—”.
Term detailAO GET /api/terms/{term_id}, GET /api/exploration/terms/{term_id}/details, GET /api/terms/{term_id}/sql-attributes
"Same student, three systems" bannerA term whose tables sit in several databases (a fact, not a match score)
CertifyAO PATCH /api/terms/{term_id} (name_certified, description_certified)
Re-runAO PUT /api/configurations/semantic-compilation {enabled: true}, which starts a compile

Ontology (step 3)

ElementSource
Object typesAO GET /api/terms
EditAO PATCH /api/terms/{term_id} (name, description)
SynonymsAO term synonyms, read only: the engine does not take synonyms over its API
Properties (backing column, status)AO GET /api/terms/{term_id}/column-attributes; certify with PATCH …/column-attributes/{attr_id}
PII flagAO column tags
Link typesAO semantic graph links
MetricsAO GET /api/terms/{term_id}/sql-attributes (expression, certified); certify with PATCH /api/sql-attributes/{attr_id}
TagsAO GET /api/tags
Actions (write tools)TB the platform's actions.yaml, written by hand; its toolsets count as writing

MCP Gateway · new server (step 4)

ElementSource
Server name, endpointTB servers/<slug>/; the endpoint is <console origin>/<platform>/<slug>/mcp (NEXT_PUBLIC_APP_URL when set), https://<host>/<platform>/<slug>/mcp on the platform's server; a console hosted elsewhere shows the server's second address, https://<host>/iblai/<platform>/<slug>/mcp
Toolsets and toolsGenerated by the host from certified AO terms and SQL attributes, the connected systems, the custom APIs and actions.yaml
Issue TokenIBL: a role and a policy per chosen toolset, made if missing, then an API key limited to those policies. The secret is shown once.
Entra / Google / custom OIDCTB authServices in tools.yaml, verified by Toolbox
Allowed origins / hostsTB flags.json, passed to Toolbox as --allowed-origins and --allowed-hosts; the gateway checks origins too
Generated config previewTB the file the host would write
Save Draft / DeployTB tools.yaml, flags.json and a new revision. Deploy also starts Toolbox and waits until it has loaded that file. Nothing is registered on ibl.ai.

MCP Gateway · server (step 5)

ElementSource
Enabled, revision, Roll BackTB flags.json and revisions/
Agent tool calls · 24h, most called toolTB gateway.log
Agents attachedIBL agents that have an MCP server registered under this server's endpoint
AttachIBL: a role and policy per chosen toolset, an API key limited to them, an MCP server registered for that agent with the key, and the agent's settings told to use it
PlaygroundThe gateway at the server's endpoint, with the signed-in user's session
Activity: agent tool callsTB gateway.log (time, tool, agent, result, duration). User and session show “—”.
Activity: runtime logTB gateway.log
TokensIBL API keys that carry an Ontology: policy (name, policies, groups, created, expires); Revoke deletes the key

Agents, Roles, Access, Catalog, Compilation, Config

ScreenSource
AgentsIBL agents and their MCP servers (auth_type, scope); call counts from TB gateway.log
RolesIBL RBAC policies named Ontology: <toolset>
AccessIBL holders of each policy, and the agents connected
CatalogAO GET /api/datasources/dbs, GET /api/schemas/{db_id}, GET /api/tables/{schema_id}, GET /api/columns/{table_id}
Compilation (was "Sync")AO GET /api/semantic-compilation/status; GET and PUT /api/configurations/{semantic-compilation,pii-detection,distinct-value-probing}. Run now is the compile switch set on.
ConfigTB files and revisions per server; AO POST /api/model/export, which needs the service key of an engine admin

Reaching the host from the browser

The browser talks to two places:

  1. ibl.ai, directly, with the SSO session. Same as the OS app.
  2. The ontology host, through this app's own route handlers (/api/host/*). They run on the platform's server, next to the engine and the Toolbox processes, behind the same TLS proxy. On a console hosted elsewhere (the pages-only build) they serve no data: each call names the platform's server in x-ontology-server, and the console forwards it to the server's second address, https://<host>/iblai/api/host/*, with the same sign-in, only to a public https address, following no redirect. The browser keeps that address per platform in localStorage["ontology_server:<platform>"] (network.md).

The route handlers keep no state of their own:

  • Every request carries the ibl.ai session (Authorization: Token <dm_token>) and its platform (x-ontology-platform: <platform key>). The handler asks ibl.ai what that session is granted on that platform and lets in a platform admin or a holder of the console grant. ibl.ai refuses a session that does not belong to the platform. A server with an engine of its own refuses every platform but the one its ONTOLOGY_PLATFORM names before it asks ibl.ai. Details in auth.md.
  • /api/host/ao/* forwards an allow-list of methods and paths to the platform's engine with its service key. The engine keeps the data.
  • /api/host/systems, /api/host/graph and /api/host/toolsets read the engine once per page load and keep that reading a few seconds.
  • /api/host/servers/* reads and writes the platform's servers/<slug>/, starts and stops the Toolbox process for that slug, rolls back a revision, and reads the gateway's log.

Agent traffic uses a second path, straight to the platform's server:

  • https://<host>/<platform>/<slug>/mcp (and the second address, https://<host>/iblai/<platform>/<slug>/mcp) is the gateway. It refuses a credential sent over plain HTTP (except on loopback), checks Origin and Host against the server's lists, asks ibl.ai what the caller's credential is granted on that platform (ibl.ai refuses a credential of another platform), limits the call to the toolsets that grant reaches, and forwards to Toolbox on 127.0.0.1:<port>. It writes only its own log. On a console hosted elsewhere, its own /<platform>/<slug>/mcp answers 404.
  • Toolbox listens on loopback only, so every call goes through the gateway. An OIDC ID token rides along in the header Toolbox expects (for example entra_token) and Toolbox verifies it.
  • When an ibl.ai agent calls the server, ibl.ai sends the API key stored in that agent's MCP server on ibl.ai; the gateway sees an ordinary API key.

Removed or replaced

WasProblemNow
Multiple "ontologies" (Institution, LMS Analytics, Advancement)no system of record; an engine is one semantic layerone ontology per ibl.ai platform, each on its own engine
DuckDB snapshot of HTTP APIs so they could be dissectedneither the engine nor Toolbox does thisAPIs are served only (Toolbox http)
"7 write operations must fail" read-only proofnot a feature of the engine, Toolbox or ibl.aithe engine's connection test; a hint to use a read-only login
97.8% cross-system match score, "accept merge"fabricated; the engine maps terms to tables, it does not scorethe fact of a term spanning N databases; certify the term
Interfaces (Person, Billable)not in the engineengine tags
Row counts, ingest progress, the engine's service lognot available over the engine's API“—” for rows; stages and log lines derived from what the engine has read
Synonym “+ add”the engine's API does not write synonymssynonyms are shown, not added
Agent analytics from ibl.ai (user, session per call)no verified endpointthe gateway's own log; user and session show “—”
p50 latency, authorized %no storeremoved
Key prefix, last used, calls per keynot in ibl.ai API keysname, policies, groups, created, expires
Rate limit per keynot settable in Toolbox or ibl.aithe gateway limits each caller to 120 requests and 30 tool calls a minute
"Claude Desktop · OIDC" row under Agentsnot an ibl.ai agentremoved; external clients show only in the Runtime Log
roles.yaml, sync-schedules.yamlartifacts of an older backendibl.ai RBAC policies; the engine's compilation settings
Masked emails in playground rowsmasking was attributed to a role policy that does not existrows as returned by the tool

Copyright © ibl.ai | support@iblai.zendesk.com