The Short Answer
The evidence that an AI agent was tested belongs to whoever owns the database the scores are written to. ibl.ai's Evals tab runs an agent against a benchmark of your own questions, has a judge model you pick score every answer, and exports every trace and score as CSV. On ibl.ai you own all the code and the data, so that evidence stays in your deployment when the model changes.
Most writing on this subject argues that agent testing is urgent because regulators are closing in. As of July this year that premise is wrong, and the correct reason is better.
The EU moved its high-risk deadline out by sixteen months. Model churn did not move at all, and model churn is what invalidates the evidence you already gathered.
What does it actually take to test an AI agent before it ships?
You run it against a benchmark: a set of test questions, each with an optional expected answer. The platform sends every question to the agent in the background and records each answer as a trace.
In ibl.ai this is the Evals tab in an agent's settings. It is not new. The tab shipped in Agentic OS in the production image releases of July 22 to 24, 2026, versions 1.104.0 through 1.108.0, recorded in the July 24 platform update.
The skill that mounts the same tab in an app built with Agentic Vibe followed in iblai/vibe v1.7.0 on July 27, 2026.
Benchmark items are typed in by hand, uploaded as a CSV, or seeded over the API from real chat traces, so recorded conversations become test cases.
The CSV rules are specific: UTF-8, a required input column, an optional expected_output column, at most 10 MB and 10,000 rows, and any row with a blank question is skipped.
Benchmarks belong to the organization, not to a single agent. That is the detail that makes comparison possible: the same 10,000-question test set can be run against several agents and scored on equal footing.
The full screen-by-screen walkthrough is in Agent Settings: Evals, and the capability deep-dive is Test an Agent Before It Ships.
Did the EU AI Act just make AI agent testing a legal requirement?
No, and anyone telling you the deadline is weeks away is working from a superseded timeline.
Regulation (EU) 2026/1744, the Digital Omnibus on AI, was adopted on 8 July 2026 and entered into force on 27 July 2026. It amends Regulation (EU) 2024/1689, the AI Act itself.
What it changed is the schedule. Obligations for Annex III standalone high-risk systems moved from 2 August 2026 to 2 December 2027. Obligations for high-risk systems embedded in regulated products, listed in Section A of Annex I, moved from 2 August 2027 to 2 August 2028.
Not everything was postponed.
The Article 50 transparency obligations were not delayed, though providers of systems generating synthetic content that were already on the market before 2 August 2026 were given until 2 December 2026 to comply with Article 50(2).
The AI Act Explorer's reading of the Omnibus sets out which articles landed where.
So the honest position is this. If your case for evaluating agents rested on an August 2026 compliance cliff, that case has sixteen more months on the clock, and the argument you were making has weakened rather than strengthened.
Build the evidence base anyway, for the reason in the next section. A deferred deadline is still a deadline, and an organization that starts gathering evaluation records in 2026 arrives at December 2027 with a history rather than a project.
Why does switching LLM providers invalidate an AI agent's test results?
Because the thing you tested is gone. An agent's behaviour is a product of its model, its prompt, its tools and its retrieved context. Change the model and you have changed the system under test, while the scores on file still describe the old one.
This is the uncomfortable corollary of being model-agnostic. The freedom to move between providers is worth real money, and it is also a standing invitation to ship an unreviewed behaviour change.
The same logic applies to the parts of an agent that are not the model at all.
We made the argument for memory in Who Owns an AI Agent's Memory When You Switch Models?: the store belongs to whoever owns the database, and the model only reads and writes it.
Evaluation scores are the same kind of object.
The practical answer is a regression check rather than a one-time certification. Keep the benchmark, re-run it after a model change or a prompt edit, and diff the two runs.
Since everything in the tab is also available over the REST API, that check can run from CI and block a prompt change from shipping, instead of waiting for a person to open a screen.
Agents can also run on a calendar. The Tasks tab takes a named prompt, a start time and a cadence of once, daily, weekly or monthly, and logs every run with its status, its start and end times and the full output.
Scheduled execution plus scheduled evaluation is what a monitored system looks like.
How does an LLM-as-judge review score an agent, and can you trust it?
A review hands every trace in a completed run to a second model, which scores it against criteria you write in plain language. You name the score, set how many traces are judged in parallel, and pick the judge provider and then the model.
The concurrency limit defaults to 4 and is capped at 20.
The judge can be a different model from the one the agent runs on, which is the entire point: an independent second opinion rather than a model grading its own homework.
Since the October 9 platform update, the judge picker checks each model's availability individually and disables the ones it cannot reach.
Every score the judge files arrives with its written reasoning for that item, so a low score explains itself rather than merely asserting itself.
Trust it partially, and on purpose. Judge models are a real and improving category, not a novelty: we covered the economics of judgment-over-generation in Generation Is Commoditized, Judgment Is the New Frontier. But a judge is a sampling instrument, not an oracle.
Which is why people can score too. From any trace, a person adds a manual score with a name, a value between 0 and 1 and an optional comment, and it sits beside the judge's scores on the same item.
One item can carry several scores from several reviews, so automated and human assessment are visible side by side rather than one standing in for the other.
The tab is admin-gated to users with permission to view the agent's analytics. Learners never see it, and it renders nothing for them even on a deep link.
What should an AI agent evaluation export contain?
One row per benchmark item, with enough identity in it to reconcile against the raw traces later.
Export CSV in ibl.ai downloads item_id, input, expected_output, actual_output and trace_id, plus one column per score. The trace_id is the column that matters most for governance, because it is what lets an auditor move from a row in a spreadsheet back to the full recorded interaction.
That shape is what makes evaluation an integrable process rather than an isolated exercise. Compliance teams pull it into a governance dashboard, engineering teams track scores over time in a BI tool, and a release process can diff two runs in a spreadsheet before a change ships.
For a broader treatment of what to measure and which metrics are worth tracking, see our guide to AI agent evaluation frameworks.
Who owns an AI agent's evaluation data?
ibl.ai is the agentic AI platform where you own all the code and the data. You self-host the entire stack inside your own perimeter, run it model-agnostic across any LLM and switch anytime, and pay by usage with no per-seat pricing — so you can deploy anywhere: your cloud, on-premise, GovCloud, or fully air-gapped.
Applied to evaluation, that resolves a problem most buyers have not priced yet. An evaluation you cannot take with you is a vendor's opinion of its own product.
Test sets encode your standards: the questions your staff actually get asked, the answers your legal counsel validated, the rubric your committee argued over.
Traces are records of what an agent told your people. Scores are the evidence that it behaved.
All three are governance artifacts with a retention life measured in years, and on ibl.ai they stay in your deployment because the stack is yours.
- You own all the code and the data. Full source code under a perpetual license, running on your infrastructure. The benchmarks, traces and scores sit in your database, not in a vendor's.
- Model-agnostic. Any LLM for the agent, any LLM for the judge, switched without rewriting the platform, which is exactly why the regression check has to be yours too.
- No per-seat pricing. Usage-based billing against a budget cap you set, so running a 10,000-item benchmark is a compute decision rather than a licensing negotiation.
- Deploy anywhere. Your cloud, your VPC, on-premise, GovCloud, or a fully air-gapped network with no outbound connectivity.
1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University.
ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on.
Want to see your agents scored against your own questions?
We will run an evaluation on a benchmark of your questions and walk through the judge's scores with you, including what a re-run looks like after a model change.