Before You Launch an Agent: Evaluation Sets, Graders and Go/No-Go Criteria

About this course

A campus agent that answers advising, IT help desk, financial aid or tutoring questions will be asked thousands of things nobody tried in the demo. This course teaches the people who decide whether such an agent is ready for students to replace "it seemed fine in the demo" with evidence: an evaluation set built from real question types, a definition of correct for each kind of question, model graders checked against human reviewers, a pass bar agreed with the business owner before the results are in, and a plan for testing after every change and monitoring after launch. It follows the Measure and Manage functions of the NIST AI Risk Management Framework (NIST AI 100-1) and its Generative AI Profile (NIST AI 600-1), the peer-reviewed research on LLM-as-a-judge, retrieval evaluation and holistic evaluation, and a statistics reference on Cohen's kappa. Labs run on paper or on the existing Evals and Grader features of the ibl.ai platform.

What you will be able to do

1. Build an evaluation set from real question types using synthetic or de-identified examples, covering typical, edge and adversarial cases, with a documented record for every item and no real student records.
2. Define what "correct" means for each question class: factual accuracy against a named source, refusal when the agent should refuse, escalation to a person, tone, and citations.
3. Score an agent with model-based graders and human review, writing rubrics that guard against the known biases of LLM judges, and use retrieval metrics such as faithfulness to diagnose grounded answers.
4. Measure agreement between a model grader and human reviewers with percent agreement and Cohen's kappa, read the direction of disagreements, and act on a low result.
5. Set a go/no-go bar with the business owner, with per-class thresholds agreed in advance and a human-reviewed sample, and write the decision memo.
6. Run regression tests when the model, prompt or sources change, and monitor the agent after launch, using the Measure and Manage functions of the NIST AI Risk Management Framework.

Who this course is for

Roles: IT, teaching and learning, student services and institutional research leaders who must decide whether a campus agent is ready for students, and the analysts, agent builders and senior staff reviewers who run the tests for them.

Institutions: colleges and universities planning or running a student-facing agent, such as an advising, IT help desk, financial aid or tutoring agent.

Before you start: no programming is needed. You should be comfortable with a spreadsheet. Have in mind one agent your campus plans to launch; the capstone is built around it.

Format

Time: 8 to 10 hours, self-paced. Level: intermediate.

Hands-on: labs in which you build a 30-item evaluation set for an illustrative advising agent, load it and run a baseline, write grader rubrics, compute agreement between two graders on a provided scored sample, and set thresholds and make a launch call. Every lab can be done on paper or on the ibl.ai platform. An AI tutor coaches each lab step by step from what you paste; it never runs anything for you.

Structure: a Start here section, four content sections of readings, worked examples, templates and labs, each ending in an AI assessment, and a capstone in which you write an evaluation plan and a go/no-go memo for one agent your campus plans to launch, followed by a reading list.

Grading: the four section AI assessments count for 60 percent of the grade and the capstone assessment for 40 percent. The pass mark is 70 percent.

Currency

Last verified 2026-11-03 against NIST AI 100-1, NIST AI 600-1, the cited research and vendor documentation, and the ibl.ai documentation generated 2026-11-03.

Evaluation practice is still changing. On the verification date NIST stated that the AI RMF 1.0 is being revised, and OpenAI's evaluation guide stated that its Evals platform becomes read-only on October 31, 2026 and is scheduled to shut down on November 30, 2026. Models, grader behavior and platform screens change. Check the current sources before relying on any page. Every scenario in the course is illustrative, and no real student records are used or allowed.