What is this course about?
Public benchmark scores tell you nothing about your workload, and most teams ship prompt changes on impression. This course builds task-specific evaluation from real traffic, covers where LLM-as-judge quietly agrees with itself, and wires a regression gate into CI so a prompt change cannot ship blind.
Who is this course for?
- ML and AI engineers
- Technical product managers owning AI features
- QA and test engineering leads
- Platform teams supporting AI delivery
What do I need before starting?
- Comfort with Python and CI pipelines
- An LLM application in development or production
What will I be able to do afterwards?
- Explain why benchmark performance does not transfer to your workload
- Build task-specific eval sets from real user traffic
- Use LLM-as-judge where it is valid and recognize where it is not
- Run human evaluation cheaply enough to sustain weekly
- Wire regression gates into CI and monitor drift in production
What does each module cover?
Why don't benchmark scores predict your results?
40 minDistribution mismatch, contamination, and the reason a leaderboard is not a procurement input.
Objectives
- Explain distribution mismatch between benchmarks and workloads
- Recognize benchmark contamination
- Determine what benchmarks are legitimately useful for
Topics
Activity. Compare a benchmark's task distribution against a sample of your real traffic.
How do you build an eval set from real traffic?
55 minSampling production queries so the eval set represents what users actually ask.
Objectives
- Sample traffic to represent the real distribution
- Include failure cases and edge cases deliberately
- Handle sensitive content in eval data
Topics
Activity. Build a 100-item eval set sampled from real traffic with documented sampling logic.
How do you build and version a golden dataset?
55 minReference answers, who writes them, and how the set stays honest over time.
Objectives
- Author reference answers with domain experts
- Version the dataset alongside the application
- Detect and retire items that no longer discriminate
Topics
Activity. Author reference answers for 30 items and measure inter-annotator agreement.
Where does LLM-as-judge quietly fail?
55 minSelf-preference, position bias, and the cases where a judge agrees with itself rather than assessing.
Objectives
- Identify the known biases in LLM judging
- Validate judge agreement against human ratings
- Determine which criteria a judge can and cannot assess
Topics
Activity. Validate an LLM judge against human ratings and measure the disagreement pattern.
How do you run human evaluation weekly?
50 minHuman evaluation designed to be cheap enough to actually sustain.
Objectives
- Design human evaluation that fits a weekly cadence
- Build rating interfaces that reduce rater load
- Calibrate raters and monitor agreement
Topics
Activity. Run a timed human evaluation round and measure cost per item.
How do you wire a regression gate into CI?
55 minBlocking a prompt or model change that degrades quality, without blocking every change.
Objectives
- Wire evaluation into the deployment pipeline
- Set thresholds that catch real degradation
- Handle intended behavior changes without disabling the gate
Topics
Activity. Wire the gate and demonstrate it blocking a degraded prompt.
How do you monitor drift in production?
50 minDetecting degradation that appears after deployment, from model updates or changing usage.
Objectives
- Instrument production quality signals
- Detect drift from model provider updates
- Alert on degradation before users report it
Topics
Activity. Instrument drift detection and simulate a provider model update.
Building the complete harness
60 minThe build module: eval set, judge, human loop, CI gate, and production monitoring assembled.
Objectives
- Assemble the full harness end to end
- Demonstrate it catching a real regression
- Document it for team handover
Topics
Activity. Assemble the harness and catch a deliberately introduced regression.
What is the capstone project?
Complete evaluation harness with a CI gate
Build a full evaluation harness for a real LLM application: traffic-sampled eval set, versioned golden dataset, validated judge, sustainable human loop, CI regression gate, and production drift monitoring.
Deliverable: A working harness with a demonstrated regression catch and handover documentation.
How are learners assessed?
- Judge validation must report the disagreement pattern against human ratings
- CI gate demonstrated blocking a deliberately degraded change
- Human evaluation cost per item measured and within a sustainable budget
What ships with the course?
Facilitator guide
Session-by-session running order, discussion prompts, and the questions that reliably derail a room.
Learner workbook
Exercises, checklists, and the templates each module's activity produces.
Hands-on lab environment
A sandboxed ibl.ai deployment so exercises run against real agents, not screenshots.
Assessment bank
Scenario questions and rubric criteria mapped to each stated learning outcome.
Source bibliography
Every primary regulation and standard cited on this page, linked and dated.
Which AI agents does this course use?
The hands-on modules run against agents already deployable on the ibl.ai platform for enterprise.
Where does the course material come from?
Every module is grounded in primary sources — the regulation, standard, or research itself, not a summary of it. Each was resolved at authoring time.
- AI Index Report
Stanford HAI
Benchmark landscape and contamination discussion for Module 1.
- AI Risk Management Framework
NIST
The Measure function, which this course operationalizes.
- Transformers documentation
Hugging Face
Evaluation tooling reference for the build modules.
- LangChain documentation
LangChain
Evaluation pipeline patterns used in Module 6.
Delivery notes
Binding guidance for anyone preparing and delivering this course.
- Module 4 must show judge failure empirically rather than asserting it. Construct a case where the judge prefers its own model's output and let participants discover it.
- Module 5's cost measurement is what determines whether human evaluation survives contact with a real team. Time it honestly; if it costs more than a few dollars per item, the design is wrong.
- The CI gate must handle intended behavior changes gracefully. A gate that fires on every deliberate improvement gets disabled within two weeks, which is worse than no gate.
- Use a real application with real traffic. Synthetic eval sets produce a course that does not transfer, because representativeness is the whole difficulty.
- Coordinate with ENT-2 — retrieval evaluation and generation evaluation overlap, and the two courses should share the harness rather than teach two.
Why run AI training on a platform you own?
You own the course, not a licence to it
Course content, learner data, and the platform run inside your perimeter — you own all the code and the data.
Model-agnostic delivery
Run the course's AI components on any LLM — Claude, GPT, Llama, Gemini, Command — and switch anytime.
No per-seat training licences
Usage-based or self-hosted, so cost tracks actual use rather than headcount.
Deploy anywhere
Cloud, private VPC, on-premise, or fully air-gapped — including for cohorts that cannot use public AI tools.
Frequently asked questions
What does the Building an LLM Eval Harness That Ships course cover?
Public benchmark scores tell you nothing about your workload, and most teams ship prompt changes on impression. This course builds task-specific evaluation from real traffic, covers where LLM-as-judge quietly agrees with itself, and wires a regression gate into CI so a prompt change cannot ship blind. It runs 6.5 hours across 8 modules across 8 modules, at advanced level, and closes with a capstone: Complete evaluation harness with a CI gate.
Who should take Building an LLM Eval Harness That Ships?
It is written for ML and AI engineers, Technical product managers owning AI features, QA and test engineering leads, Platform teams supporting AI delivery. Prerequisites: Comfort with Python and CI pipelines; An LLM application in development or production.
Can we run this course on our own infrastructure?
Yes. ibl.ai is model-agnostic and deploy-anywhere — cloud, private VPC, on-premise, or fully air-gapped — and you own all the code and the data. Cohort data, submissions, and any material learners upload stay inside your perimeter, which matters for enterprise teams that cannot send work to a public AI tool.
How do we get access to Building an LLM Eval Harness That Ships?
Request access and we will set it up for your cohort — hosted by ibl.ai, or running against your own deployment. Tell us the group size and timing you need, and whether it should run inside your own perimeter.
How much does AI training for enterprise cost on ibl.ai?
There is no per-seat pricing — you pay for usage or self-host and pay only for the infrastructure, so a 5,000-person rollout does not cost 5,000 licences. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University.