Chapter 6Routing, Not Choosing

The previous chapter argued that models are commodities and that the architecture must treat them as settings. This one takes the next step. Once switching is free, there is no reason to switch everything at once. A sovereign platform routes each request to the model that suits it, and changes the routing as the market changes. This chapter is about what that routing considers, and what it looks like from an administrator's chair.

Four dimensions of a routing decision

Every request an agent makes to a model can be scored on four axes, and a routing policy is a rule over those axes.

Cost. Providers price per million tokens, input and output separately, and the spread between the cheapest capable model and the most expensive is two orders of magnitude. A platform that knows the price of each model and the expected size of each request can estimate the cost of a decision before making it.

Latency. A model behind a chat window has a few seconds; a model behind a voice call has a few hundred milliseconds; a model running a nightly batch job has all night. Smaller models are faster. Models hosted nearby are faster than models across an ocean. The right choice for a voice agent is wrong for a research summary.

Capability. Some requests need long context, tool calling, image understanding, or reasoning across many steps. Most need none of these. The routing policy sends the hard requests to models that can handle them and the easy requests to models that are cheap.

Data classification. This is the axis the other platforms do not have, and the one that makes routing a sovereignty tool rather than a cost optimization. A request that contains student records, patient data, privileged legal material or classified information may be allowed to reach only certain models: those running on the institution's own hardware, or those under a contract that permits it, or none at all outside the perimeter.

A routing rule is a sentence like: "Requests from the admissions agents go to the institution's own open-weight model if they contain applicant data, otherwise to the cheapest commercial model that passes the admissions evaluation set, with the frontier model as a fallback if the first attempt fails its own check."

Routing by data classification

The fourth axis deserves its own section because it is where policy becomes enforceable.

Most institutions already classify data. Public, internal, confidential, restricted; or some local equivalent. Most AI deployments ignore that classification entirely, because the model is fixed and the only choice is whether to use it. A routing layer changes that. The connector that fetches data can tag it with its classification. The agent's context carries the tag. The router refuses to send restricted-tagged context to any model not on the restricted-approved list.

This turns a governance document into a system behaviour. The policy that says "patient data does not leave the network" is no longer a training slide; it is a rule the router applies to every request, and an audit line records every decision. When the CISO asks whether any protected data has reached an external model in the last quarter, the answer is a query, not an investigation.

The approved list for restricted data will usually be short: models running on the institution's own GPUs, which Chapter 7 covers, and perhaps a commercial provider under a specific contract in a specific region. That is fine. Restricted workloads are a minority, and the six-month rule means the open-weight models that can run locally are good enough for most of them.

Fallbacks and outages

Commercial model providers have outages, rate limits and quiet deprecations. A platform with one provider inherits all of them. A routing layer with a fallback chain does not.

The simplest useful policy is a primary model, a secondary from a different provider, and a tertiary running locally. A request that fails at the primary, with a timeout, a rate-limit response or a content refusal that the platform recognizes as a provider-side policy rather than a real problem, retries at the secondary without the user noticing. If every external provider is down, the local model answers, perhaps less well, but the institution's services do not stop.

The same chain handles deprecation. When a provider retires a model, the platform's catalogue marks it, the evaluation suite runs against the proposed replacement, and the routing rule is updated. Agents built against the standard interface never know.

Open-weight models in the mix

Open-weight models change the routing calculation in two ways beyond data classification.

They have no per-token price. The cost is hardware and electricity, which means the marginal cost of one more request is close to zero once the hardware is bought. For high-volume, low-stakes workloads, a classifier running on every incoming email, a summarizer running on every meeting, a local model can be dramatically cheaper than any commercial API, and the routing policy should send that traffic there by default.

They are stable. A commercial model can change behaviour overnight when the provider updates it. An open-weight model pinned to a specific version behaves identically until the institution decides otherwise. For workloads where consistency matters more than peak capability, regulated reporting, anything a human has signed off on, that stability is worth more than a few benchmark points.

The practical pattern in most sovereign deployments is a mix: one or two commercial providers for the hard problems and the long tail, one or two open-weight models on local hardware for the volume, the restricted data and the fallback. The routing layer makes the mix invisible to the people using the agents and visible to the people paying for them.

What the administrator sees

None of this matters if it requires an engineer to operate. The routing layer has to be a surface an administrator can use.

In practice that means a model catalogue with live prices and capabilities, per provider. A per-agent setting that chooses the default model, with the evaluation results beside each option. A classification policy table that maps data tags to allowed models. A fallback chain, in order. A spend view that shows tokens and cost by agent, by model and by day, against the budget cap the institution set. And an audit log, filterable by classification tag, that shows which model saw which request.

An administrator with that surface can run the institution's model strategy as an operating function: review the catalogue monthly, re-run the evaluations when something new appears, move traffic when the numbers justify it, and answer the governance questions with data. That is what "model-agnostic" means when it is a property of the system rather than a line in a brochure. The next chapter turns to the hardware the local half of that mix runs on.