Course description
Predictive analytics in higher education flag students as at risk at scale: more than 500 universities used one vendor's risk scores alone (The Markup, 2021), yet a 2026 review of 28 studies found no definitive evidence that early-alert systems work as implemented (Maeda et al., preprint). This course audits the system you already have rather than teaching you to build one. You will test whether your alerts change outcomes, compute error rates by race, ethnicity, Pell status and first-generation status the way the AERA Open study did (among students who went on to graduate, models predicted failure for 21 percent of Hispanic and 19 percent of Black students, against 12 percent of White and 6 percent of Asian students), apply the FERPA school-official test to every data flow and vendor, rewrite alert wording and the student-notice policy, and put a logged human-review gate in front of every action a flag triggers. Why now: the effectiveness evidence has matured, the equity findings are published, and alert volumes are large enough that an unreviewed flag is a liability rather than a service.
About this course
Predictive analytics in higher education flag students as at risk at scale: more than 500 universities used one vendor's risk scores alone (The Markup, 2021), yet a 2026 review of 28 studies found no definitive evidence that early-alert systems work as implemented (Maeda et al., preprint). This course audits the system you already have rather than teaching you to build one. You will test whether your alerts change outcomes, compute error rates by race, ethnicity, Pell status and first-generation status the way the AERA Open study did (among students who went on to graduate, models predicted failure for 21 percent of Hispanic and 19 percent of Black students, against 12 percent of White and 6 percent of Asian students), apply the FERPA school-official test to every data flow and vendor, rewrite alert wording and the student-notice policy, and put a logged human-review gate in front of every action a flag triggers. Why now: the effectiveness evidence has matured, the equity findings are published, and alert volumes are large enough that an unreviewed flag is a liability rather than a service.
What you will be able to do
- State, with sources, what the evidence shows about early-alert effectiveness, and identify the study design (randomized trial, regression discontinuity, systematic review) behind each claim the vendor or the campus makes.
- Compute false-positive and false-negative rates by subgroup from a risk-score export joined to an outcomes file, and report them in a subgroup error-rate table with base rates alongside.
- Apply the FERPA school-official test to every data flow feeding the model and every vendor receiving data, and document the three conditions in 34 CFR 99.31 for each vendor.
- Rewrite alert wording so a flag states an observation, its source and a next step rather than a verdict, and draft the institution's student-notice policy.
- Design and log a human-review gate that records who reviewed a flag, what they decided and why, and state the staffing the gate requires at the institution's alert volume.
- Produce an audit report a provost or faculty senate can act on, including a recommendation to keep, change or retire each predictor.
Who this course is for
Roles: vice presidents and deans of student success, institutional research and analytics directors, advising directors and advising-technology leads, privacy officers and FERPA compliance staff, registrars, and faculty senate members who sit on data-governance committees.
Institutions: any two-year or four-year institution that runs a vendor early-alert platform or an in-house risk model, and system offices that license one platform for several campuses.
Before you start: familiarity with your institution's alert workflow (who raises a flag, who receives it, what happens next) and the ability to read a table of rates. No coding and no statistics beyond percentages are required. The lab uses an AI agent through a browser on a supplied synthetic export.
Format
- Estimated time: 10 to 12 hours.
- Level: intermediate. Assumes you have seen your institution's alerts in use; assumes nothing about modelling.
- Hands-on: one lab on a supplied synthetic export (3,000 students with risk score, alert history, race and ethnicity, Pell status, first-generation status and term outcomes, plus a 12-clause sample vendor contract excerpt and three alert message templates), five templates, three discussions, and a capstone on the learner's own alert data.
- Structure: 4 sections, each ending in an AI conversation assessment, then a capstone assessment.
Self-paced. Every lesson is taught by an AI tutor that opens with a short story and works through one idea and one question at a time. Each section ends with an AI conversation assessment, and the course closes with a capstone assessment. Section assessments count for 60% and the capstone for 40%; 70% passes. There are no multiple-choice quizzes.
Currency of this course
Last verified 2026-09-28. No FERPA regulatory change affecting this course was found in 2025-26 when it was prepared; federal guidance on AI in student-success analytics is still moving, so Unit 3.2.3 teaches where to check and every audit report records its check date.
Outline
Section 0 · Start here
- 0.1 Orientation
Section 1 · Does it work? Reading the evidence before auditing the model
- 1.1 What the studies show
- 1.2 Where alerts work
- 1.3 Section 1 assessment
Section 2 · The error-rate audit by subgroup
- 2.1 What the models get wrong, and for whom
- 2.2 Running the audit
- 2.3 Section 2 assessment
Section 3 · FERPA boundaries per data flow and vendor
- 3.1 The school-official test
- 3.2 Mapping the data flows
- 3.3 Section 3 assessment
Section 4 · Alert wording, student notice and the human-review gate
- 4.1 What the flag says and who is told
- 4.2 The human-review gate
- 4.3 Section 4 assessment
Section 5 · Capstone & wrap-up
- 5.1 Capstone
- 5.2 Capstone assessment
- 5.3 Wrap-up