ibl.ai's agent Evals tab runs an agent against a benchmark of test questions, has a judge model you pick score every answer against criteria you write, lets people add 0–1 scores by hand, and exports every trace and score as CSV, all from the agent's own settings, on a platform where you own all the code and the data.
This is a deep-dive into a shipped capability, not a launch. The Evals tab first shipped in Agentic OS in the production image releases of July 22–24, 2026 (1.104.0 → 1.108.0), recorded in the July 24 platform update.
The /iblai-vibe-agent-evals skill that mounts the same tab in any app built with Agentic Vibe followed in iblai/vibe v1.7.0 on July 27, 2026.
As of the October 9 platform update, the judge picker checks each model's availability individually and disables unavailable ones (@iblai/web-containers 1.35.1).
How do you test an AI agent before it goes live?
You run it against a benchmark. In ibl.ai, open the agent's settings and choose Evals. Pick a benchmark, click New Evaluation, and the platform sends every question in it to the agent in the background, recording each answer as a trace.
The table lists each run with its status, who started it and when.
A run moves through pending → in progress → completed; its badge is green when completed, red when failed and blue while still in flight.
Each row's menu has five actions: View results, New review, Check status, Export CSV and Delete.
View results, New review and Export CSV stay disabled until the run completes. A run name is optional — leave it blank and the platform generates one, such as the run-091c4937 in the screenshot.
What goes into an agent benchmark?
A benchmark is a set of test questions, each with an optional expected answer. Benchmarks belong to the organization, not to one agent, so the same test set can be run against several agents and compared on equal footing. Each run, by contrast, is scoped to the agent it tested.
Manage benchmarks creates one from a name and a description, then fills it with question/answer pairs two ways: typed in by hand, or uploaded as a CSV. The selector picks the first benchmark automatically, and New Evaluation stays disabled until one exists.
The CSV rules are on the screen: UTF-8, a required input column for the question, an optional expected_output column for the answer, at most 10 MB and 10,000 rows, and any row with a blank question is skipped. A sample file downloads from the same dialog.
Over the API, benchmark items can also be seeded from existing chat traces — the input and the agent's response are copied from each trace — so real conversations become test cases.
How does an LLM-as-judge review score an AI agent?
A review hands every trace in a completed run to a second model, which scores it against your criteria. Choose New review and the Evaluate dialog asks for four things.
Criteria is a rubric in plain language — the placeholder suggests accuracy, completeness and clarity, with accuracy weighted most.
Score name is the label the scores are filed under, such as quality. Max concurrency, 4 by default and capped at 20, sets how many traces are judged in parallel. The fourth is the judge model.
The judge is chosen provider first, then model. The organization in this screenshot offered ten providers, from Amazon and Anthropic to NVIDIA and xAI; yours lists whatever your organization has configured.
The judge can differ from the model the agent runs on, which is how you get an independent second opinion. Every score the judge files comes with its written reasoning for that item, so a low score explains itself.
Scores are not only the judge's. From any trace, a person can add a manual score — a name, a value between 0 and 1 and an optional comment — and it sits beside the judge's scores on the same item. One item can carry several scores from several reviews.
What does an agent evaluation export contain?
Export CSV downloads one row per benchmark item, with item_id, input, expected_output, actual_output and trace_id, plus one column per score. That is enough to diff two runs in a spreadsheet, or to feed a regression check that blocks a prompt change from shipping.
Everything in the tab is also available over the platform's REST API: benchmarks and their items (including CSV upload), runs, judge reviews, scores and the export.
So an evaluation can run from CI or from another agent, not only from a person at a screen. The API side is packaged as the iblai-api-agent-eval skill.
Who can run agent evaluations in ibl.ai?
Administrators and agent builders — anyone with permission to view the agent's analytics.
Learners never see it: for them the tab renders nothing, even when someone deep-links to it. An app can also show benchmarks in the organization's settings, as a Benchmarks tab under AI & data.
The full walkthrough, screen by screen, is in Agent Settings: Evals. Agents can also do their own work on a calendar — see agents that work on a schedule.
Want to see your agents scored against your own questions?
We'll run an evaluation on a benchmark of your questions and walk through the judge's scores with you. Book a 30-minute demo or talk to the ibl.ai team — ibl.ai is family-owned and operated from New York, NY.