# RAG on Enterprise Knowledge: Architecture, Chunking, Evals > Enterprise · AI Course · ENT-2 > Source: https://ibl.ai/solutions/enterprise/course/rag-on-enterprise-knowledge > Last updated: 2026-08-25 **Production retrieval over enterprise content — chunking strategy, hybrid search, permission-aware retrieval, and the eval harness that proves it works.** ## The Short Answer **RAG fails in production on permission-aware retrieval and evaluation, not on embedding choice. ibl.ai builds retrieval that inherits each user's permissions and ships with a gold-set eval harness, running where you own all the code and the data — so your document corpus is never uploaded to a vendor's index you cannot inspect.** On ibl.ai you own all the code and the data, run it model-agnostic across any LLM, and pay with no per-seat pricing — so you can deploy anywhere, from your own cloud to a fully air-gapped network. [Request Access](https://ibl.ai/contact) · [Explore Enterprise](https://ibl.ai/solutions/enterprise) ## Course facts - **Level:** Advanced - **Duration:** 7.5 hours across 8 modules - **Format:** Technical workshop with a build lab - **Modules:** 8 - **Catalog code:** ENT-2 - **Frameworks covered:** NIST AI RMF, SOC 2, ISO 27001 ## What is this course about? RAG demos well and fails in production for reasons that are predictable and fixable. This course covers chunking by document type, hybrid search and reranking, and the requirement most implementations skip entirely — permission-aware retrieval. It ends with a gold-set eval harness wired into CI so a retrieval change cannot ship blind. ## Who is this course for? - ML and data engineers building retrieval systems - Enterprise architects - Knowledge management technical leads - Platform teams supporting internal AI ### What do I need before starting? - Comfort with Python and APIs - Basic understanding of embeddings and vector search ## What will I be able to do afterwards? - Choose chunking strategies by document type rather than by default - Combine dense, sparse, and reranking retrieval and know what each fixes - Implement permission-aware retrieval that respects the requesting user - Build a gold-set evaluation before tuning anything - Wire retrieval evaluation into CI as a regression gate ## What does each module cover? ### Module 1 — Why does naive RAG demo well and fail in production? The gap between a curated demo corpus and a real enterprise document estate. _(45 min)_ **Objectives** - Characterize how production corpora differ from demo corpora - Predict the failure modes each difference produces - Set evaluation criteria before building **Topics:** Corpus messiness · Duplication and staleness · Access heterogeneity · Failure prediction **Activity:** Sample 50 documents from a real corpus and categorize what would break naive retrieval. ### Module 2 — How should you chunk each document type? Policy, contract, ticket, wiki, and spreadsheet each need different treatment. _(60 min)_ **Objectives** - Select chunking strategy by document structure - Preserve tables and structured content - Handle very long and very short documents **Topics:** Structure-aware chunking · Table handling · Length extremes · Overlap strategy **Activity:** Chunk five document types three ways each and compare retrieval on a shared question set. ### Module 3 — What do embeddings, keyword search, and reranking each fix? Hybrid retrieval, and the specific failure each component addresses. _(55 min)_ **Objectives** - Explain what dense retrieval misses and sparse retrieval catches - Configure hybrid fusion - Apply reranking where it earns its latency cost **Topics:** Dense retrieval limits · Sparse and keyword search · Fusion strategies · Reranking economics **Activity:** Measure each component's contribution independently on your question set. ### Module 4 — How do you make retrieval permission-aware? The requirement most implementations skip, and the reason enterprise RAG projects get halted. _(60 min)_ **Objectives** - Implement retrieval filtered by the requesting user's permissions - Handle permission changes without a full reindex - Test systematically for cross-boundary leakage **Topics:** Permission filtering · ACL synchronization · Reindex avoidance · Leakage testing **Activity:** Implement permission filtering and run a cross-user leakage test suite. ### Module 5 — What do you do about contradictory and stale documents? Duplication, superseded policy, and the confident wrong answer they produce. _(50 min)_ **Objectives** - Detect near-duplicate and superseded content - Rank by authority and recency - Surface contradiction rather than blending it **Topics:** Duplicate detection · Authority ranking · Recency weighting · Contradiction surfacing **Activity:** Seed contradictions into a corpus and verify the system surfaces rather than averages them. ### Module 6 — How do you build a gold set before tuning? Evaluation data first, because tuning without it is guesswork with extra steps. _(55 min)_ **Objectives** - Build a gold set from real user traffic - Recruit domain experts to verify answers - Version the gold set as the corpus changes **Topics:** Traffic-derived questions · Expert verification · Gold set versioning · Coverage analysis **Activity:** Build a 50-item gold set from real queries with expert-verified answers. ### Module 7 — Which retrieval metrics predict user trust? The metrics that correlate with whether people keep using the system. _(50 min)_ **Objectives** - Select metrics that track user-perceived quality - Measure groundedness and citation accuracy - Detect the metrics that improve while trust falls **Topics:** Recall and precision at k · Groundedness · Citation accuracy · Metric-trust divergence **Activity:** Correlate your metrics against a human quality rating on the same queries. ### Module 8 — Wiring evaluation into CI as a regression gate The build module: an eval harness that blocks a retrieval change from shipping blind. _(60 min)_ **Objectives** - Wire the gold set into a CI pipeline - Set regression thresholds that catch real degradation - Handle expected changes without disabling the gate **Topics:** CI integration · Threshold setting · Expected change handling · Gate discipline **Activity:** Wire the gate and prove it blocks a deliberately degraded configuration. ## What is the capstone project? **Production RAG pipeline with a CI regression gate.** Build a complete retrieval pipeline over a real corpus with type-appropriate chunking, hybrid retrieval, permission-aware filtering, contradiction surfacing, and a versioned gold set wired into CI as a blocking gate. _Deliverable:_ A deployed pipeline, a versioned gold set, and a CI gate demonstrated blocking a bad change. ## How are learners assessed? - Cross-user leakage suite must pass with zero leakage - Gold set performance against a pre-agreed threshold - CI gate demonstrated blocking a deliberately degraded configuration ## What ships with the course? - **Facilitator guide.** Session-by-session running order, discussion prompts, and the questions that reliably derail a room. - **Learner workbook.** Exercises, checklists, and the templates each module's activity produces. - **Hands-on lab environment.** A sandboxed ibl.ai deployment so exercises run against real agents, not screenshots. - **Assessment bank.** Scenario questions and rubric criteria mapped to each stated learning outcome. - **Source bibliography.** Every primary regulation and standard cited on this page, linked and dated. ## Which AI agents does this course use? - [Knowledge Agent](https://ibl.ai/solutions/enterprise/agent/knowledge-agent) - [IT Help Desk Agent](https://ibl.ai/solutions/enterprise/agent/it-help-desk-agent) - [Operations Agent](https://ibl.ai/solutions/enterprise/agent/operations-agent) - [Data Analysis Agent](https://ibl.ai/solutions/enterprise/agent/data-analysis-agent) ## Where does the course material come from? Every module is grounded in primary sources — the regulation, standard, or research itself, not a summary of it. Each was resolved at authoring time. - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) — Lewis et al., arXiv. The foundational RAG architecture paper. - [Embeddings guide](https://platform.openai.com/docs/guides/embeddings) — OpenAI. Reference for embedding behavior and dimensionality trade-offs. - [LlamaIndex](https://www.llamaindex.ai/) — LlamaIndex. Reference implementations for structure-aware chunking patterns. - [LangChain documentation](https://python.langchain.com/docs/introduction/) — LangChain. Retrieval and reranking pipeline patterns used in Module 3. ## Delivery notes Binding guidance for anyone preparing and delivering this course: - Module 4 is what makes this an enterprise course rather than a RAG tutorial. Permission-aware retrieval is the reason enterprise projects get halted at security review, and it deserves the most build time. - Do not tune before Module 6. Sequencing evaluation after tuning is the mistake the course exists to correct, and reordering the modules would undermine its central discipline. - Use a genuinely messy corpus with duplicates, superseded policies, and inconsistent formatting. A clean corpus produces a course that does not transfer. - Keep the stack swappable. Teaching one vendor's retrieval product contradicts the model-agnostic position and dates the course to that product's roadmap. - Ship the leakage test suite as a reusable artifact — it is the piece security teams will ask for and engineering teams will not build. ## Why run AI training on a platform you own? - **You own the course, not a licence to it.** Course content, learner data, and the platform run inside your perimeter — you own all the code and the data. - **Model-agnostic delivery.** Run the course's AI components on any LLM — Claude, GPT, Llama, Gemini, Command — and switch anytime. - **No per-seat training licences.** Usage-based or self-hosted, so cost tracks actual use rather than headcount. - **Deploy anywhere.** Cloud, private VPC, on-premise, or fully air-gapped — including for cohorts that cannot use public AI tools. ## Frequently asked questions ### What does the RAG on Enterprise Knowledge: Architecture, Chunking, Evals course cover? RAG demos well and fails in production for reasons that are predictable and fixable. This course covers chunking by document type, hybrid search and reranking, and the requirement most implementations skip entirely — permission-aware retrieval. It ends with a gold-set eval harness wired into CI so a retrieval change cannot ship blind. It runs 7.5 hours across 8 modules across 8 modules, at advanced level, and closes with a capstone: Production RAG pipeline with a CI regression gate. ### Who should take RAG on Enterprise Knowledge: Architecture, Chunking, Evals? It is written for ML and data engineers building retrieval systems, Enterprise architects, Knowledge management technical leads, Platform teams supporting internal AI. Prerequisites: Comfort with Python and APIs; Basic understanding of embeddings and vector search. ### Can we run this course on our own infrastructure? Yes. ibl.ai is model-agnostic and deploy-anywhere — cloud, private VPC, on-premise, or fully air-gapped — and you own all the code and the data. Cohort data, submissions, and any material learners upload stay inside your perimeter, which matters for enterprise teams that cannot send work to a public AI tool. ### How do we get access to RAG on Enterprise Knowledge: Architecture, Chunking, Evals? Request access and we will set it up for your cohort — hosted by ibl.ai, or running against your own deployment. Tell us the group size and timing you need, and whether it should run inside your own perimeter. ### How much does AI training for enterprise cost on ibl.ai? There is no per-seat pricing — you pay for usage or self-host and pay only for the infrastructure, so a 5,000-person rollout does not cost 5,000 licences. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University. ## More Enterprise courses - [Agentic AI for the Enterprise: From Chatbot to Workforce](https://ibl.ai/solutions/enterprise/course/agentic-ai-for-the-enterprise): What separates an agent from a chatbot — tools, memory, autonomy — and the orchestration patterns that let agents finish multi-step work without supervision. - [The Enterprise AI Cost Model: Per-Seat vs Token vs Owned](https://ibl.ai/solutions/enterprise/course/enterprise-ai-cost-model): Model AI spend across pricing shapes at real headcount — where per-seat licensing breaks, what tokens actually cost, and when owning the stack wins. - [AI Security: The OWASP LLM Top 10 in Production](https://ibl.ai/solutions/enterprise/course/ai-security-owasp-llm-top-10): Securing deployed LLM systems — prompt injection, data leakage, supply chain, and excessive agency — with the controls and tests for each. - [AI Governance in Practice: NIST AI RMF, ISO 42001, EU AI Act](https://ibl.ai/solutions/enterprise/course/ai-governance-nist-iso-eu-ai-act): Operationalize three overlapping frameworks into one governance program — inventory, risk classification, controls, and the evidence auditors ask for. - [Building an LLM Eval Harness That Ships](https://ibl.ai/solutions/enterprise/course/llm-eval-harness-that-ships): Move from vibes to measurement — task-specific eval design, LLM-as-judge and its limits, regression gates, and production monitoring. - [Model Context Protocol: Connecting Agents to Enterprise Systems](https://ibl.ai/solutions/enterprise/course/model-context-protocol-enterprise): MCP as the integration layer for enterprise agents — server design, authentication, authorization, and exposing internal systems without exposing them to everyone.