Policy Answers Staff Can Trust: Permission-Aware AI Search over Policies, Handbooks and Catalogs

About this course

A policy chatbot answers the questions staff and faculty ask every day (leave, the tuition benefit, grade-change deadlines, catalog rules), and it is only useful if the answer is the current one, the one that applies to the asker, and one the asker was allowed to see. Duke HR's Ask CHRIS logged 900+ queries in its first two weeks with a 2 percent thumbs-down rate, and still invented the names of campus eateries (Duke Chronicle, 2026-06-08). This course teaches the operating model, not the software stack: which document is authoritative and who owns it, effective dates and catalog years, retrieval that returns only what the asker may see, the line between general policy answers and answers that use the employee's own record, citations with a decline-and-route path to a human, keeping the index current as policies change, and measuring quality with faithfulness and recall scores plus thumbs-down rates. You will build a permission-aware policy agent on a sample handbook set, write its evaluation set, and leave with the registers, templates and thresholds to run one at your institution.

What you will be able to do

1. Produce an authoritative-document register for at least three policy domains that names the owner, the effective-date rule and the retirement trigger for each document, and lists every duplicate or superseded copy to exclude from the index.
2. Specify permission-aware retrieval for a policy corpus (which roles may retrieve which documents, how the index enforces it, and what the retrieval log records) using OWASP LLM08:2025 as the checklist.
3. Distinguish a general-policy answer from a personal answer that uses the asker's department, tenure or record, and state the data-access and FERPA conditions each requires.
4. Compute faithfulness, context recall and answer relevance on an evaluation set of at least 60 questions, and set a release threshold and a thumbs-down target for the tool.
5. Write a decline-and-route rule set: the conditions under which the agent refuses to answer, what it says, and which office receives the question.
6. Run a change-driven re-indexing process with a freshness service level, and demonstrate on the sample corpus that a superseded policy stops appearing in answers within that window.

Who this course is for

Roles: HR directors and HR service-center leads, policy office and university policy coordinators, registrars and catalog editors, IT service desk managers, staff in the general counsel's office who own policy text, and the academic-technology or IT staff asked to build or buy the tool.

Institutions: any college or university, and especially institutions with more than one entity (a university and its health system, or a system with several campuses) where the same question has a different answer depending on who is asking.

Before you start: knowing which offices own the policies at your institution, and comfort with a spreadsheet. No coding is required. The labs use an AI agent through a browser and a supplied sample corpus.

Format

• Estimated time: 10 to 12 hours.
• Level: intermediate. Assumes you answer or own policy questions; assumes nothing about software or machine learning.
• Hands-on: two labs on a supplied sample corpus (an employee handbook of about 80 policies for the two entities of a fictional university and its health system, a 2025-26 and a 2026-27 catalog excerpt, 12 registrar procedures, and 60 evaluation questions with reference answers, 10 of which have no answer in the corpus), five templates, three discussions, and a capstone on the learner's own corpus.
• Structure: 4 sections, each ending in an AI conversation assessment, then a capstone assessment.

Self-paced. Every lesson is taught by an AI tutor that opens with a short story and works through one idea and one question at a time. Each section ends with an AI conversation assessment, and the course closes with a capstone assessment. Section assessments count for 60% and the capstone for 40%; 70% passes. There are no multiple-choice quizzes.

Currency of this course

Last verified 2026-09-28. The frameworks cited here (NIST AI 600-1, the OWASP Top 10 for LLM applications) are voluntary and are revised on their own schedules; FERPA itself is unchanged, but how an indexed corpus touches education records must be re-checked whenever the corpus changes.