--- title: "AI Agent Security Is an Infrastructure Problem, Not a Feature" slug: "ai-agent-security-infrastructure-problem-not-a-feature" author: "ibl.ai Engineering" date: "2026-08-02 12:00:00" category: "Premium" topics: "ai agents, ai security, enterprise ai, agent governance, self-hosted ai, sovereign ai, prompt injection, audit logging" summary: "Uber's security lead says securing AI agents is what keeps him up at night, and Google just shipped agent evaluation tooling to production. The tooling layer is maturing; the infrastructure question underneath it is not. This post explains why you cannot fully secure an agent whose reasoning runs on someone else's servers, and gives the five-question perimeter test to run on any agent platform before you sign." banner: "" thumbnail: "" linkedin: | Uber's security lead said the quiet part out loud this week: "AI agents are everywhere at Uber. The thing that keeps me up at night is how we are going to secure them." Uber published a full agent security framework. Google made Agent and Model Evaluations generally available in its Gemini Enterprise platform. The tooling layer is maturing fast. The layer underneath it is not, and that's the one that decides the outcome. Here's the structural problem. An AI agent isn't a static endpoint — it's a reasoning loop with system access. It reads data, calls APIs, and takes actions on its own. A compromised agent doesn't just leak; it acts. Now ask where that reasoning happens. On most enterprise deployments today, it happens on a vendor's servers, over a vendor's API, inside a vendor's security perimeter. You cannot fully secure what you do not control. Five questions worth asking before you sign anything (we call it the perimeter test): 1. Where does the agent reason — our infrastructure or yours? 2. Do we receive the full source code? 3. Can we swap the underlying model without rewriting our security controls? 4. Does every tool call, data access, and decision produce an auditable log we hold? 5. Can we enforce our own access policies, or only the ones you expose? Any "no" means you're not securing your agents. You're trusting someone else to. And the old excuse for accepting that trade — that frontier-class reasoning only exists behind a cloud API — expired. DeepSeek released V4 Flash 0731 as open weights under MIT this week, scoring 50 on the Artificial Analysis Intelligence Index at roughly 28x cheaper output tokens. Frontier-class reasoning now runs behind your own firewall. The cost barrier to self-hosted AI collapsed. The security argument for it just got much stronger. Full breakdown, including what infrastructure-first agent security actually requires: https://ibl.ai/blog/ai-agent-security-infrastructure-problem-not-a-feature #iblai #AgenticAI #EnterpriseAI #AISecurity #DataSovereignty --- ## The Short Answer **AI agent security is decided by infrastructure, not by features, because an agent is not a static endpoint — it is a reasoning loop with production access that reads data, calls APIs, and takes actions autonomously. A compromised agent does not merely leak information; it acts on the compromised instruction. You cannot fully secure a reasoning loop that runs on someone else's servers.** The controls that matter — where inference executes, who can read the prompts and weights, whether every tool call is logged in a system you hold, whether you can enforce your own access policy — are all properties of the deployment, not items on a vendor's feature list. That is why the strongest agent security posture is structural: self-hosted execution inside your perimeter, model-agnostic architecture so no single provider's breach is automatically yours, and complete audit trails you own. ## Why is AI agent security an infrastructure problem rather than a feature? Because the agent's attack surface is its reasoning, and reasoning happens wherever the inference runs. Traditional application security assumes a fixed set of endpoints with predictable inputs and outputs. An agent breaks that assumption: it decides which tools to call, in what order, on data it retrieves at runtime. The consequence is that a prompt-injection payload buried in a support ticket or a PDF is not merely a content problem — it is potentially an instruction to a system with credentials. Uber's security lead put it plainly this week: "AI agents are everywhere at Uber. The thing that keeps me up at night is how we are going to secure them." Uber published a comprehensive agent security framework alongside that remark — not because the problem is solved, but because it is outgrowing the available solutions. The tooling is catching up. Google made Agent and Model Evaluations generally available in its Gemini Enterprise Agent Platform in the same week, giving teams a way to measure, test, and monitor agents in development and production. That is real progress at the observability layer. It does not answer the layer beneath it. Evaluation tells you how an agent behaves; it does not tell you who else can see the data it reasoned over, or whose incident response covers the machine it ran on. ## What does the typical enterprise agent security failure look like? It follows a three-phase arc that repeats across industries, and the damage is done in the gap between phase one and phase three. **Phase one: build and ship.** Teams deploy agents to prove return on investment. The question is capability — can this handle support tickets, triage IT requests, draft contracts, summarize case notes? Speed wins. Security is a checkbox on the launch review. **Phase two: discover the attack surface.** Every deployed agent is a new entry point with credentials attached. Unlike a form or an API route, it makes decisions. Teams find that the same agent that summarizes a document will happily follow an instruction embedded in that document unless something stops it. **Phase three: retrofit.** Rate limiting, output filtering, permission scoping, audit logging — all necessary, all significantly harder to add after the agent is embedded in a workflow that a business unit now depends on. Retrofitting permission scoping in particular tends to break the very integrations that justified the project. The organizations avoiding this arc are not the ones with better filters. They are the ones that made the deployment decision — where the agent runs and who holds the code — before phase one. ## What can't you secure when the agent runs on a vendor's infrastructure? More than most security reviews acknowledge, because the questions that matter are answered by the deployment model rather than by the vendor's certifications. A SOC 2 report describes the vendor's controls; it does not give you the controls.
Security question Managed SaaS agent Self-hosted agent
Where does the reasoning execute? Vendor infrastructure Your servers, your network
Who can read prompts and retrieved context? Vendor staff, per vendor policy Only your privileged users
Who holds the audit log? Vendor, exported on their schema You, in your SIEM
Blast radius of a vendor breach Your data, disclosed on their timeline Contained to your perimeter
Model change Vendor's roadmap decision Your configuration change
Policy enforcement Limited to exposed settings Anything the source code allows
The right-hand column is not a longer feature list. It is the same list of questions answered by ownership instead of by trust. The distinction becomes concrete during an incident: forensic reconstruction of an agent's tool calls requires logs at a granularity most managed platforms never expose, and a subpoena or breach notification runs on the log holder's timeline, not yours. The same dependency logic applies further down the stack, as we covered in [supply-chain attacks and why infrastructure ownership contains them](https://ibl.ai/blog/supply-chain-attacks-ai-security-own-infrastructure). ## What does infrastructure-first agent security actually require? Five controls, each of which is a deployment property before it is a product capability. **Self-hosted execution.** The agent reasons inside your security perimeter — your servers, your network segments, your encryption keys. Sensitive data never crosses a boundary you do not administer, and the attack surface reduces to infrastructure you already monitor. **Model agnosticism as a security control.** Binding an agent framework to one provider's models means inheriting that provider's security posture wholesale; a vulnerability or outage there is yours by construction. A model-agnostic platform lets you swap models without touching your controls, and run open-weight models locally when sensitivity demands it. **Complete audit trails.** Every tool call, data access, and decision logged at forensic granularity in a store you own. Compliance is the easy case; the hard case is incident response, where you need to reconstruct what the agent did and why, months later. **Role-based agent permissions.** Agents inherit the permissions of the user or service they act for — an HR agent cannot reach financial systems, a support agent cannot read personnel records. Obvious in principle, and treated as an afterthought by most agent frameworks, which default to a single service credential with broad scope. **Input and output filtering with programmable rails.** Screen inputs before they reach the model and filter outputs before they reach users or tools. This matters most for agents with system access, where an injected instruction can trigger a real-world action. ibl.ai implements this layer with NVIDIA NeMo Guardrails — programmable rails, jailbreak and injection defense, PII redaction, and network isolation — described in detail on our [NemoClaw service page](https://ibl.ai/service/nemoclaw). ## Does self-hosting frontier-class models still cost more than an API? Not the way it did a year ago, and that is what changes the security calculus. This week DeepSeek released V4 Flash 0731 as open weights under an MIT license, scoring 50 on the Artificial Analysis Intelligence Index — frontier-class territory — at roughly 28x cheaper output tokens than comparable closed models. The practical implication for security teams is direct: the standard objection to self-hosting was that keeping data inside the perimeter meant accepting materially weaker reasoning. That trade is largely gone. Organizations can now run models of comparable capability on their own hardware, behind their own firewalls, under their own policies. That does not make every workload a self-hosting candidate. It does mean the decision reverts to where it belongs — a data-sensitivity and compliance judgment per workload — instead of being forced by a capability gap. The [open-weight releases arriving on a monthly cadence](https://ibl.ai/blog/open-weight-ai-enterprise-nemotron-3-ultra-2026) keep moving the line in the same direction. ## What should you ask an AI agent vendor before signing? Run the perimeter test. Five questions, and any "no" tells you the security model is trust rather than control. 1. **Can this run entirely on our infrastructure** — our cloud account, our VPC, on-premise, or air-gapped? 2. **Do we receive the full source code**, under a license that lets our engineers read and modify it? 3. **Can we swap the underlying model** — commercial or open-weight — without changing our security controls? 4. **Does every agent interaction generate an auditable log** we hold, at tool-call granularity? 5. **Can we enforce our own access policies**, or are we limited to the settings the vendor exposes? A vendor can answer yes to all five only if the architecture was built for it. Retrofitting deployment ownership onto a managed product is not a roadmap item; it is a different product. For organizations that need every answer to be yes, ibl.ai is an Agentic AI Operating System deployed on your own infrastructure — managed cloud, VPC, on-premise, or fully [air-gapped](https://ibl.ai/service/air-gapped-ai) — with full source code and data ownership and any model you choose. It is family-owned and operated from New York, NY, which matters to government, defense, and regulated buyers evaluating who ultimately controls the vendor as well as the software. In a world where agents hold production access to your most sensitive systems, the difference between securing them and trusting someone else to secure them is the whole of your risk position. --- *ibl.ai is an Agentic AI Operating System that organizations deploy on their own infrastructure with full source code and data ownership — model-agnostic, usage-based, and deployable anywhere from managed cloud to fully air-gapped. Family-owned and operated from New York, NY. [Learn more about enterprise deployment](https://ibl.ai/solutions/enterprise).*