Blog
LLM Infrastructure
Model selection, hosting, fine-tuning, cost optimization, and scaling LLM-powered systems in production.
775 articles in this category
How Washington Made Sovereign AI the Path of Least Resistance
The White House AI framework exempts open-weight models from review entirely. Regulation has accidentally made self-hosted AI the lowest-friction path for organizations that need to move fast.
Goldman Sachs Runs AI Coding Agents With 12,000 Engineers
Goldman Sachs is running hundreds of AI coding agents alongside 12,000 human engineers — in production, not demos. The moat isn't the model. It's the harness: eval, routing, governance, audit trails.
Nemotron 3.5 Lightning and NeMo Switchyard: Why Agents Need an Open Routing Layer
NVIDIA released Nemotron 3.5 Lightning (30B total, 3B active) and NeMo Switchyard, an open routing library. Together they make the model the cheapest part of an agent deployment — and move the value to the routing layer. Here is what enterprises should own, and the cost math for routing by task.
On-Premise Foundation Models: Which Vendors Allow It
Which foundation model vendors actually permit on-premise deployment, sorted into open-weight, contracted-private, and API-only tiers — and why picking a model vendor is not the same decision as picking the platform that runs it.
How Universities Are Building AI Infrastructure They Actually Own
Per-seat AI licensing charges a 15,000-user campus $340K–$1.02M a year for access it never owns. Here is what the alternative looks like in production, with Syracuse University's published registration-season numbers and the cost math at campus scale.
K-12 AI Agent Governance Can't Be Borrowed from Enterprise
Districts are adopting enterprise AI governance templates wholesale, and the templates were written for a population that can consent. This post maps each enterprise control to why it fails for minors, sets out grade-band guardrail requirements, and reads the Kimi K3 sandbox escape for what it means on a school network.
Why 73% of Enterprise AI Budgets Break — and the Fix
The FinOps Foundation's 2026 survey of 1,192 practitioners found 73% of enterprises overshot their AI cost projections. The cause is procurement shape, not model prices: per-seat licenses bill headcount while agents multiply token spend invisibly. This post shows the math at 2,000 seats and the enforcement layer — caps, per-agent budgets, cost-aware routing — that actually holds a budget.
AI Governance Platforms: Enterprise Buyer's Guide for 2026
How enterprises should evaluate AI governance platforms in 2026: model inventory, NIST AI RMF and EU AI Act risk tiering, policy enforcement, and the ownership question that decides where your compliance evidence actually lives.
Karnataka's Government-First AI Test: Capability Without Dependency
Karnataka made sovereign data residency a precondition, not a clause — and that single sequencing choice is what separates buying AI capability from buying a dependency. The five-question procurement test, with the per-seat cost math at 5,000 to 500,000 government users.
AI's Price Spread Has to Rationalize. Hedge Both Ways.
A million tokens costs about $26 from one frontier lab and about $0.50 from a Chinese provider — a 52x spread for capability now 3–6 months apart. Spreads that wide close, and buyers cannot know which direction. The only position that survives either outcome is one where the model is a component you can swap.
Tokenmaxxing: The AI Bill Your CFO Cannot See
One operator reports token costs doubling every 45 days against a 5% productivity gain. Usage-based pricing is the right shape for AI spend — but only if you can see the meter, attribute it to a workflow, and switch the model underneath. Here is what to instrument before the variance lands in an earnings call.
Shadow AI in Healthcare: The Patient Safety Crisis
Clinicians are already pasting PHI into consumer AI tools, and no acceptable-use policy has ever stopped a productivity habit. The fix is infrastructure: a sanctioned AI platform the hospital owns and runs itself, so PHI never leaves the building.
Shadow AI Is Already Inside Every Government Agency
Unsanctioned AI use is already routine across federal agencies, and in government the exposure is statutory rather than commercial — Privacy Act records sent to commercial providers, federal records generated in systems the agency cannot subpoena, supply-chain restrictions under EO 13873, and mosaic classification spillage. This post maps each exposure to its legal basis and gives the data-classification tiers that decide which workloads need managed cloud, agency-controlled infrastructure, or a fully air-gapped deployment.
The Open-Weight Tipping Point: Two 2-Trillion-Parameter Models
Two models above 2 trillion parameters became available as open weights in a single week: Moonshot's Kimi K3 at 2.8T with a 1M-token context, and Alibaba's Qwen 3.8-Max at 2.4T with 95B active per token. This post does the memory arithmetic on what it actually takes to serve models that size, prices the alternatives, and explains why the durable advantage is model-agnostic infrastructure rather than any single model.
AI Agent Security Is an Infrastructure Problem, Not a Feature
Uber's security lead says securing AI agents is what keeps him up at night, and Google just shipped agent evaluation tooling to production. The tooling layer is maturing; the infrastructure question underneath it is not. This post explains why you cannot fully secure an agent whose reasoning runs on someone else's servers, and gives the five-question perimeter test to run on any agent platform before you sign.
Q2 2026 Earnings: AI Infrastructure Pays — For Whoever Owns It
The quarter ending June 30, 2026 settled the question of whether AI infrastructure pays off: AWS grew 37% to $42.2B, Google Cloud 82% to $24.8B, Azure crossed $100B annualized, and Copilot passed 30 million paid seats. This post does the arithmetic on what those seats cost a 10,000-person enterprise versus token-priced and self-hosted alternatives, and shows where the return actually lands.
SaaS Fragmentation Is the Hidden Cost of Enterprise AI
Enterprises run six or seven per-seat tools that each hold a partial copy of the same customer. That fragmentation, not model capability, is what stalls AI deployment — and it carries a per-seat bill that grows with headcount. This post itemizes the fragmentation tax and shows the MCP-based orchestration layer that reads across every system instead of adding another one.
AI Budgets Are Growing 40% a Year. Deployment Isn't.
Enterprise AI investment is compounding near 40% a year — roughly double cloud and mobile at the same stage — yet most of it never reaches production. This post introduces deployment yield, the ratio of AI budget attached to systems real users touch, and shows why the missing control plane, not model capability, is what security and compliance actually block on.
The AI Harness Thesis: Orchestration Beats Model Selection
Enterprises spend their AI strategy debating which model to buy. The model is the commodity — it is replaced every few months and its price falls. The harness around it (retrieval, validation, routing, memory) is the durable asset, and it only compounds if you own it.
Self-Hosted Voice AI Agents for Hospital Health Systems
What it actually costs to run outbound voice AI agents on hospital-owned infrastructure, which BAAs you still need, and where PHI travels during an AI phone call.
The Semantic Layer AI Agents Need — and Who Should Own It
A warehouse semantic layer gives dashboards consistent metrics; AI agents need that plus an operational layer — actions, permissions, audit — with governance. ibl.ai ships both as one open-source, MIT-licensed ontology you self-host and own.
Ontology vs Taxonomy vs Knowledge Graph: What AI Needs
A taxonomy classifies things into a hierarchy; an ontology adds typed relationships, attributes, and actions; a knowledge graph is the ontology populated with your real data. AI agents need all three levels — and you should own the whole stack.
How to Build an Organizational Ontology: A Practical Guide
A practical, step-by-step guide to building an organizational ontology: model the nouns, add the verbs, connect your systems once over MCP, govern access by role, and ship the whole layer from a CLI — open source, self-hosted, and owned by you.
What Is a Data Ontology? Definition, Layers, and Examples
A data ontology is a structured, machine-readable map of your organization's entities, relationships, and actions that AI agents reason over. Definition, the two layers, ontology vs database schema, a concrete cross-system example — and an open-source, self-hosted implementation you own.
About LLM Infrastructure
Running large language models in production requires careful infrastructure planning—from model selection and hosting to fine-tuning, cost optimization, and GPU provisioning. Explore practical guides on building reliable, scalable LLM infrastructure that balances performance, cost, and latency for real-world applications.