# How to Self-Host an LLM in Production > Source: https://ibl.ai/resources/guides/self-host-an-llm-in-production > Last updated: 2026-08-19 *From hardware sizing to serving, evaluation and monitoring — including the honest answer on when self-hosting is cheaper than an API and when it is not* Reading time: 16 min read | Difficulty: advanced **On ibl.ai you own all the code and the data, run it model-agnostic across any LLM, and pay with no per-seat pricing — so you can deploy anywhere, from your own cloud to a fully air-gapped network.** ## How do you self-Host an LLM in Production? Self-hosting converts a variable per-token cost into a fixed capacity cost. That trade is straightforwardly bad at low utilization and straightforwardly good at high utilization, and the entire decision turns on which side of that line your workload sits. The engineering is no longer exotic. Mature serving stacks handle batching, quantization and KV caching well, and open-weight models cover a large share of real enterprise work. What still catches teams out is everything around the model: evaluation that makes model choice defensible, routing that sends only the right work to the local model, and monitoring that catches quality regression after an update. ## Prerequisites - **A measured workload profile:** Tokens per request, requests per hour and peak concurrency taken from real traffic or a representative sample. Without these, hardware sizing is guesswork. - **GPU capacity or a path to it:** Owned hardware, or reserved cloud GPU capacity. Memory determines which models fit; utilization determines whether the economics work. - **An evaluation set from your own domain:** A held-out set of real tasks with known-good outputs. This is what turns 'the model seems fine' into a defensible model-selection decision. - **Somewhere to put logs and metrics:** Internal observability, since a self-hosted deployment intended for privacy reasons should not ship prompts to a third-party monitoring service. ## Step 1: Decide what actually needs to run locally Self-hosting everything is rarely correct. The workloads that belong on local hardware are high-volume, routine, or privacy-constrained; hard reasoning on low volume usually belongs on a frontier API. - [ ] Classify workloads by volume, difficulty and data sensitivity - [ ] Identify the high-volume routine tail — This is where local hosting pays. - [ ] Identify anything touching regulated data — This runs locally regardless of economics. - [ ] Leave the hardest low-volume work on a frontier API ## Step 2: Size hardware from the measured profile Model weights plus KV cache at your peak concurrency determine the memory requirement. Compute determines throughput once memory fits, not before. - [ ] Calculate weight memory: parameters times bytes per parameter after quantization - [ ] Add KV cache for peak concurrent sequences at your context length - [ ] Leave headroom for fragmentation and growth - [ ] Validate the estimate with a load test before committing to procurement **Tips:** - Context length drives KV cache linearly. Reducing retrieved context to what the task needs cuts memory and cost together. ## Step 3: Choose and quantize the model against your evaluation set Quantization is usually what makes a capable model fit affordable hardware. How much quality it costs is workload-specific and must be measured rather than assumed. - [ ] Shortlist open-weight models that fit your memory budget - [ ] Evaluate at full precision and at 4-bit on your own set - [ ] Record the quality delta explicitly — It belongs in the model's provenance record. - [ ] Pin the chosen model and quantization by hash ## Step 4: Stand up the serving layer A production serving stack handles continuous batching, paged attention and streaming. These are what turn a model that runs into a model that serves concurrent users at acceptable latency. - [ ] Enable continuous batching for throughput under concurrency - [ ] Configure maximum context and concurrency limits deliberately - [ ] Set up health checks and graceful degradation under load - [ ] Load-test at peak concurrency before going live ## Step 5: Put routing in front of it The local model should receive the work it handles well and nothing else. Routing is what makes self-hosting an optimization rather than a quality regression. - [ ] Define routing rules by task type — Start static; dynamic difficulty estimation adds complexity for marginal gain. - [ ] Route anything containing regulated data to the local model unconditionally - [ ] Define a fallback path when the local model is saturated - [ ] Log which model served each request — Required for provenance and for cost attribution. ## Step 6: Monitor quality, not just uptime Availability monitoring will not catch the failure that matters. A model update, a quantization change or a shift in input distribution degrades quality while every dashboard stays green. - [ ] Re-run the evaluation set on a schedule and after every change - [ ] Track latency at the tail, not the mean - [ ] Sample real outputs for human review - [ ] Alert on evaluation score regression, not only on errors ## Common Mistakes ### Self-hosting everything **Consequence:** The hardest low-volume tasks run on a model that handles them poorly, and the deployment gets judged on its worst output. **Prevention:** Route by task type from day one. Local for high-volume, routine and privacy-constrained work; frontier API for the hardest requests. ### Sizing from vendor guidance rather than measurement **Consequence:** Hardware is bought for a workload nobody characterized, and is either idle or saturated within a quarter. **Prevention:** Measure tokens per request and peak concurrency on real traffic before any procurement decision. ### Skipping the evaluation set **Consequence:** Model choice cannot be defended, quantization quality loss is invisible, and regressions after updates go undetected. **Prevention:** Build a held-out set of real domain tasks before selecting a model, and re-run it after every change. ### Shipping prompts to a third-party monitoring service **Consequence:** A deployment self-hosted for privacy reasons leaks the exact content it was built to protect, through the observability layer. **Prevention:** Keep logging and evaluation internal, and audit every observability integration for what it transmits. ## FAQ **Q: When is self-hosting cheaper than a hosted API?** Above a utilization threshold. Self-hosting converts variable per-token cost into fixed capacity cost, so the question is whether you can keep the hardware busy. High-volume routine workloads cross over far sooner than bursty exploratory ones. **Q: How much GPU memory does a model need?** Parameters times bytes per parameter after quantization, plus KV cache for your peak concurrent sequences at your context length. A 32GB card comfortably serves a 27–34B model in 4-bit quantization. **Q: Does quantization hurt quality?** Sometimes, and how much is workload-specific. Measure it on your own evaluation set at full precision and at 4-bit rather than relying on general benchmarks, and record the delta as part of the model's provenance. **Q: Do you need a dedicated ML team to self-host?** Not necessarily, but you need someone who owns capacity planning, updates, evaluation and incident response. That is a real cost line, and comparisons that omit it flatter the self-hosted option. **Q: Can you mix self-hosted and hosted models?** Yes, and it is the configuration most mature deployments settle on. Local open-weight models take the high-volume and privacy-constrained work while a frontier API handles the hardest requests, with routing deciding per request. ## Can you do this on infrastructure you own? **ibl.ai is the agentic AI platform where you own all the code and the data. You self-host the entire stack inside your own perimeter, run it model-agnostic across any LLM and switch anytime, and pay by usage with no per-seat pricing — so you can deploy anywhere: your cloud, on-premise, GovCloud, or fully air-gapped.** - **You own all the code and the data.** Full source code under a perpetual license, running on your infrastructure. Not API access to someone else's platform — the stack itself is yours. - **Model-agnostic.** Run any LLM — Claude, GPT, Gemini, Llama, Command, or your own fine-tune — and switch providers without rewriting the platform. - **No per-seat pricing.** Usage-based billing against a budget cap you set. Cost tracks what your organization actually uses, not how many people you employ. - **Deploy anywhere.** Your cloud, your VPC, on-premise, GovCloud, or a fully air-gapped network with no outbound connectivity. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University. ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on.