--- title: "Cache Side-Channels Break the On-Premise Assumption" slug: "cache-side-channel-attack-local-llm-on-premise-isolation" author: "Jaione Amigot" date: "2026-09-11 11:00:00" category: "Premium" topics: "cache side-channel, LLM security, on-premise AI, HIPAA, tenancy isolation, host isolation, healthcare AI security, air-gapped AI" summary: "A USENIX Security 2025 paper reconstructed a local LLM's output from CPU cache patterns at a 5.2% edit distance, using unprivileged code on the same host. Air-gapping closes the network boundary, not the host one." banner: "" thumbnail: "" linkedin: | Hospitals moved AI on-premise for HIPAA. "On our servers" and "invisible to every other process on those servers" are not the same claim. At USENIX Security 2025, researchers from the Chinese Academy of Sciences reconstructed both the input and the output text of a locally-running LLM by watching CPU cache access patterns. Output came back at a 5.2% average edit distance from ground truth, input at 17.3%. Three things worth getting right about it. It is not new. The paper went up in May 2025 and was presented that August. A 2026 follow-up extended the idea from CPU cache to the CPU/GPU shared cache on Apple Silicon. It requires co-residency, not privilege. The attacker runs unprivileged code on the same machine, sharing a last-level cache. It never talks to the victim's model. And it is not a defeat for air-gapping. Air-gapping is a claim about the network boundary. This attack lives entirely inside the host boundary. An internet-isolated cluster that schedules an inference pod next to arbitrary tenant workloads reproduces the threat model inside your own data center. So the controls that decide this are not network controls: → Tenancy — dedicated inference hosts, not a shared scheduling pool → Who can execute code on those hosts at all, including containers, which do not partition cache → Whether model weights are memory-mapped across security domains → Hardware isolation: cache partitioning, and confidential computing where the throughput cost is acceptable None of these are questions a managed-API buyer is in a position to ask. That is the actual argument for self-hosting — not that it is automatically safer, but that the control exists and can be exercised. With ibl.ai you own all the code and the data — self-hosted inside your own perimeter, model-agnostic across any LLM, usage-based with no per-seat pricing, deployable anywhere from your own cloud to a fully air-gapped network. #iblai #AgenticAI #EnterpriseAI #HealthcareAI #HIPAA #InfoSec --- ## The Short Answer **Researchers at USENIX Security 2025 reconstructed a local LLM's output text from CPU cache access patterns alone, at a 5.2% average edit distance, using unprivileged code on the same machine. Air-gapping closes the network boundary, not the host boundary. With ibl.ai you own all the code and the data, so you control tenancy, host access and process isolation — the controls that actually decide this.** Running a model on your own servers and running it where no other process can observe it are two different guarantees, and only the first one comes for free. ## What did the USENIX Security 2025 cache side-channel paper actually recover? Both the input and the output text of a locally-running LLM, at high fidelity. The paper is [*I Know What You Said: Unveiling Hardware Cache Side-Channels in Local Large Language Model Inference*](https://arxiv.org/abs/2505.06738), by Zibo Gao and seven co-authors at the Institute of Information Engineering, Chinese Academy of Sciences. It was posted to arXiv on 10 May 2025 and presented at the 34th USENIX Security Symposium that August. Two leaks are chained together. Token *values* come from cache access patterns in the token-embedding lookup: a cache hit on row *j* of the embedding weight matrix indicates that the token with index *j* was processed. Token *positions* come from the timing of the autoregressive decoding steps. A reconstruction algorithm fuses the two signals, and a fine-tuned model restores fluent text from the recovered token list. The reported accuracy is specific. Restored output text had an **average edit distance of 5.2%** from ground truth and restored input text **17.3%**, with average cosine similarity above 98% for both. The evaluation covered Llama, Gemma, Falcon, Mistral and Phi across ten inference frameworks. One correction to how this circulates: it is not a discovery from this week. The core result has been public for sixteen months and peer-reviewed for over a year. ## Does the cache side-channel attack on local LLMs require privileged access or co-residency? Co-residency yes, privilege no. That combination is the entire security story. The attacker has to execute unprivileged code on the same physical machine as the inference process, sharing a last-level cache with it. It never interacts with the victim's LLM — no prompt, no API call, no network path to the model at all. It also needs the model weights to be physically shared memory. Mainstream frameworks memory-map model files for zero-copy loading, so attacker and victim touch the same physical pages. OS page deduplication is the fallback route to the same condition, and the measurement primitive is Flush+Reload. One condition matters a great deal for defenders: the token-embedding lookup has to run on the CPU. That held in nine of the ten frameworks tested — only HuggingFace Transformers kept the embedding on the GPU, and that was the single configuration in which the attack failed. The authors state their own limits clearly. Input reconstruction degrades as inputs get longer, GPU-side embedding is out of reach without a separate GPU cache attack, and the attack assumes the tokenizer and model file format are known. They also report having disclosed the issues to the affected software developers. ## Why doesn't air-gapping an on-premise LLM prevent a cache side-channel attack? Because air-gapping is a statement about the network boundary, and this attack lives entirely inside the host boundary. The framing worth discarding is that air-gapping fails here against physics. Nothing is defeated by physical law. The attack needs an adversary's process scheduled on the same silicon as the inference process, which is an access-control fact about the host, not an inevitability. That distinction is what makes it actionable. An air-gapped host running one workload, with no untrusted code and no interactive access for anyone outside the admin group, does not satisfy the paper's threat model. An internet-isolated Kubernetes cluster that schedules the inference pod onto a shared node pool alongside arbitrary tenant workloads satisfies it precisely — inside the covered boundary, behind every firewall. The boundary also keeps moving. [SLAC](https://arxiv.org/abs/2608.09075), posted in August 2026 and accepted to ACM CCS 2026, extends the idea to the system-level cache that CPU and GPU share on Apple silicon, again from an unprivileged co-resident process. It recovers input keywords at up to 94.8% and response tokens at up to 88.9% — but those headline numbers come from its GPU-priming variant, and its LLM results are on TinyLlama-1.1B and GPT-2 Medium. The authors state the remaining conditions plainly: a restricted vocabulary of the 3,000 most frequent tokens, domain dictionaries of 100 to 1,000 keywords, and results averaged over 1,000 to 4,000 runs on an M1. They propose no concrete mitigation beyond cache-hierarchy redesign. ## Which controls actually reduce cache side-channel exposure on an inference host? Four, and none of them is a network control. **Tenancy.** Dedicated inference hosts rather than a shared scheduling pool. This is the control that does the most work, because it removes the co-residency the attack depends on. **Who can execute code on the host.** Containers are not a cache boundary. Namespaces and cgroups partition filesystems, processes and CPU shares; they do not partition a last-level cache. A container scheduled beside the inference container is co-resident in the sense that matters here. **How model weights are loaded.** Disabling zero-copy memory-mapped loading removes the shared-page condition, at a cost the authors measured as **17% slower model loading, and about 32% extra memory once two or more instances run concurrently**. Restricting page sharing to designated inference sessions is the lower-cost version of the same idea. **Hardware isolation.** Cache partitioning through Intel Cache Allocation Technology isolates the last-level cache, though the paper notes it is absent from the consumer-grade hardware where many local deployments run. Confidential computing goes further and costs more. [One 2025 measurement](https://arxiv.org/abs/2505.16501) of H100 confidential-computing mode reports non-CC latency **20–30% lower** and non-CC throughput **45–70% higher** than confidential mode, in a single-GPU batch-inference configuration with model swapping. Where the framework allows it, moving the embedding lookup onto the GPU also removes the specific signal this attack reads. ## Does this mean self-hosting an LLM is less safe than using a managed AI API? No, and the paper does not support that reading. Every control above is a decision about a host you operate, and a managed API buyer cannot make any of them. They cannot specify tenancy, enumerate what else runs on the inference node, pin workloads, or verify any of it after the fact. The provider may run a well-isolated fleet. The customer's evidence for that is an attestation document. The asymmetry is not that self-hosting is automatically safer. It is that self-hosting is the only arrangement in which the question has an answer you can check. That is the same argument that makes an [air-gapped clinical AI platform](/blog/air-gapped-clinical-ai-platform) survive a third HIPAA review, applied one layer below the network diagram. It comes with an obligation. Ownership of the host is what makes host-level isolation possible, and it is also what makes it your responsibility. A self-hosted deployment on a shared, multi-tenant node pool with broad developer shell access has the ownership and none of the isolation. ## How does ibl.ai isolate inference hosts for regulated deployments? By putting the whole stack on infrastructure the institution controls, so tenancy and host access are configuration rather than a vendor policy. With ibl.ai you own all the code and the data. The platform self-hosts inside your own perimeter with full source code access — so the loading path, the scheduling and the isolation posture are all inspectable rather than asserted — is model-agnostic across any LLM and switchable at any time, is usage-based with no per-seat pricing, and can deploy anywhere from your own cloud to on-premise, GovCloud, or a fully air-gapped network. For a security reviewer, the practical consequence is that the questions in this post are answerable. You can state which hosts run inference, what else is scheduled on them, who can execute code there, and how weights are loaded — then verify each answer against a system you operate. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University. ibl.ai is family-owned and operated from New York, NY. *Related reading: [air-gapped clinical AI platform](/blog/air-gapped-clinical-ai-platform) — what the network boundary does buy a HIPAA-covered health system, and where it stops.* *Sources: the attack, threat model, accuracy figures and mitigations from [Gao et al., USENIX Security 2025](https://arxiv.org/abs/2505.06738); the CPU-to-GPU extension and its stated conditions from [SLAC](https://arxiv.org/abs/2608.09075); the confidential-computing overhead measurement from [Martínez Ibarra et al.](https://arxiv.org/abs/2505.16501).* ## Why does owning the AI stack matter? **ibl.ai is the agentic AI platform where you own all the code and the data. You self-host the entire stack inside your own perimeter, run it model-agnostic across any LLM and switch anytime, and pay by usage with no per-seat pricing — so you can deploy anywhere: your cloud, on-premise, GovCloud, or fully air-gapped.** - **You own all the code and the data.** Full source code under a perpetual license, running on your infrastructure. Not API access to someone else's platform — the stack itself is yours. - **Model-agnostic.** Run any LLM — Claude, GPT, Gemini, Llama, Command, or your own fine-tune — and switch providers without rewriting the platform. - **No per-seat pricing.** Usage-based billing against a budget cap you set. Cost tracks what your organization actually uses, not how many people you employ. - **Deploy anywhere.** Your cloud, your VPC, on-premise, GovCloud, or a fully air-gapped network with no outbound connectivity. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University. ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on.