What is this course about?
RAG demos well and fails in production for reasons that are predictable and fixable. This course covers chunking by document type, hybrid search and reranking, and the requirement most implementations skip entirely — permission-aware retrieval. It ends with a gold-set eval harness wired into CI so a retrieval change cannot ship blind.
Who is this course for?
- ML and data engineers building retrieval systems
- Enterprise architects
- Knowledge management technical leads
- Platform teams supporting internal AI
What do I need before starting?
- Comfort with Python and APIs
- Basic understanding of embeddings and vector search
What will I be able to do afterwards?
- Choose chunking strategies by document type rather than by default
- Combine dense, sparse, and reranking retrieval and know what each fixes
- Implement permission-aware retrieval that respects the requesting user
- Build a gold-set evaluation before tuning anything
- Wire retrieval evaluation into CI as a regression gate
What does each module cover?
Why does naive RAG demo well and fail in production?
45 minThe gap between a curated demo corpus and a real enterprise document estate.
Objectives
- Characterize how production corpora differ from demo corpora
- Predict the failure modes each difference produces
- Set evaluation criteria before building
Topics
Activity. Sample 50 documents from a real corpus and categorize what would break naive retrieval.
How should you chunk each document type?
60 minPolicy, contract, ticket, wiki, and spreadsheet each need different treatment.
Objectives
- Select chunking strategy by document structure
- Preserve tables and structured content
- Handle very long and very short documents
Topics
Activity. Chunk five document types three ways each and compare retrieval on a shared question set.
What do embeddings, keyword search, and reranking each fix?
55 minHybrid retrieval, and the specific failure each component addresses.
Objectives
- Explain what dense retrieval misses and sparse retrieval catches
- Configure hybrid fusion
- Apply reranking where it earns its latency cost
Topics
Activity. Measure each component's contribution independently on your question set.
How do you make retrieval permission-aware?
60 minThe requirement most implementations skip, and the reason enterprise RAG projects get halted.
Objectives
- Implement retrieval filtered by the requesting user's permissions
- Handle permission changes without a full reindex
- Test systematically for cross-boundary leakage
Topics
Activity. Implement permission filtering and run a cross-user leakage test suite.
What do you do about contradictory and stale documents?
50 minDuplication, superseded policy, and the confident wrong answer they produce.
Objectives
- Detect near-duplicate and superseded content
- Rank by authority and recency
- Surface contradiction rather than blending it
Topics
Activity. Seed contradictions into a corpus and verify the system surfaces rather than averages them.
How do you build a gold set before tuning?
55 minEvaluation data first, because tuning without it is guesswork with extra steps.
Objectives
- Build a gold set from real user traffic
- Recruit domain experts to verify answers
- Version the gold set as the corpus changes
Topics
Activity. Build a 50-item gold set from real queries with expert-verified answers.
Which retrieval metrics predict user trust?
50 minThe metrics that correlate with whether people keep using the system.
Objectives
- Select metrics that track user-perceived quality
- Measure groundedness and citation accuracy
- Detect the metrics that improve while trust falls
Topics
Activity. Correlate your metrics against a human quality rating on the same queries.
Wiring evaluation into CI as a regression gate
60 minThe build module: an eval harness that blocks a retrieval change from shipping blind.
Objectives
- Wire the gold set into a CI pipeline
- Set regression thresholds that catch real degradation
- Handle expected changes without disabling the gate
Topics
Activity. Wire the gate and prove it blocks a deliberately degraded configuration.
What is the capstone project?
Production RAG pipeline with a CI regression gate
Build a complete retrieval pipeline over a real corpus with type-appropriate chunking, hybrid retrieval, permission-aware filtering, contradiction surfacing, and a versioned gold set wired into CI as a blocking gate.
Deliverable: A deployed pipeline, a versioned gold set, and a CI gate demonstrated blocking a bad change.
How are learners assessed?
- Cross-user leakage suite must pass with zero leakage
- Gold set performance against a pre-agreed threshold
- CI gate demonstrated blocking a deliberately degraded configuration
What ships with the course?
Facilitator guide
Session-by-session running order, discussion prompts, and the questions that reliably derail a room.
Learner workbook
Exercises, checklists, and the templates each module's activity produces.
Hands-on lab environment
A sandboxed ibl.ai deployment so exercises run against real agents, not screenshots.
Assessment bank
Scenario questions and rubric criteria mapped to each stated learning outcome.
Source bibliography
Every primary regulation and standard cited on this page, linked and dated.
Which AI agents does this course use?
The hands-on modules run against agents already deployable on the ibl.ai platform for enterprise.
Where does the course material come from?
Every module is grounded in primary sources — the regulation, standard, or research itself, not a summary of it. Each was resolved at authoring time.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis et al., arXiv
The foundational RAG architecture paper.
- Embeddings guide
OpenAI
Reference for embedding behavior and dimensionality trade-offs.
- LlamaIndex
LlamaIndex
Reference implementations for structure-aware chunking patterns.
- LangChain documentation
LangChain
Retrieval and reranking pipeline patterns used in Module 3.
Delivery notes
Binding guidance for anyone preparing and delivering this course.
- Module 4 is what makes this an enterprise course rather than a RAG tutorial. Permission-aware retrieval is the reason enterprise projects get halted at security review, and it deserves the most build time.
- Do not tune before Module 6. Sequencing evaluation after tuning is the mistake the course exists to correct, and reordering the modules would undermine its central discipline.
- Use a genuinely messy corpus with duplicates, superseded policies, and inconsistent formatting. A clean corpus produces a course that does not transfer.
- Keep the stack swappable. Teaching one vendor's retrieval product contradicts the model-agnostic position and dates the course to that product's roadmap.
- Ship the leakage test suite as a reusable artifact — it is the piece security teams will ask for and engineering teams will not build.
Why run AI training on a platform you own?
You own the course, not a licence to it
Course content, learner data, and the platform run inside your perimeter — you own all the code and the data.
Model-agnostic delivery
Run the course's AI components on any LLM — Claude, GPT, Llama, Gemini, Command — and switch anytime.
No per-seat training licences
Usage-based or self-hosted, so cost tracks actual use rather than headcount.
Deploy anywhere
Cloud, private VPC, on-premise, or fully air-gapped — including for cohorts that cannot use public AI tools.
Frequently asked questions
What does the RAG on Enterprise Knowledge: Architecture, Chunking, Evals course cover?
RAG demos well and fails in production for reasons that are predictable and fixable. This course covers chunking by document type, hybrid search and reranking, and the requirement most implementations skip entirely — permission-aware retrieval. It ends with a gold-set eval harness wired into CI so a retrieval change cannot ship blind. It runs 7.5 hours across 8 modules across 8 modules, at advanced level, and closes with a capstone: Production RAG pipeline with a CI regression gate.
Who should take RAG on Enterprise Knowledge: Architecture, Chunking, Evals?
It is written for ML and data engineers building retrieval systems, Enterprise architects, Knowledge management technical leads, Platform teams supporting internal AI. Prerequisites: Comfort with Python and APIs; Basic understanding of embeddings and vector search.
Can we run this course on our own infrastructure?
Yes. ibl.ai is model-agnostic and deploy-anywhere — cloud, private VPC, on-premise, or fully air-gapped — and you own all the code and the data. Cohort data, submissions, and any material learners upload stay inside your perimeter, which matters for enterprise teams that cannot send work to a public AI tool.
How do we get access to RAG on Enterprise Knowledge: Architecture, Chunking, Evals?
Request access and we will set it up for your cohort — hosted by ibl.ai, or running against your own deployment. Tell us the group size and timing you need, and whether it should run inside your own perimeter.
How much does AI training for enterprise cost on ibl.ai?
There is no per-seat pricing — you pay for usage or self-host and pay only for the infrastructure, so a 5,000-person rollout does not cost 5,000 licences. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University.