AI Tutoring Research: What the RCTs Show and How to Deploy a Tutor That Improves Learning

About this course

AI tutoring research is now large enough to mislead: a decision-maker can find a randomized trial to support almost any position, from "outperforms active learning" to "harms exam scores." This course teaches you to read the trials as a body of evidence, benchmark them against human tutoring, and design a deployment that follows what the studies actually found. You will learn why unguarded GPT-4 raised practice scores by 48 percent and cut unassisted exam scores by 17 percent in the PNAS trial (2025), why the Harvard physics study's 0.73 to 1.3 standard-deviation effect covers two lessons in one course with expert-written prompts, and why the strongest higher-education trial to date is exactly that two-lesson Harvard study, with the rest of the rigorous evidence coming from K-12 and international settings. No higher-education meta-analysis of generative-AI tutoring had been located at verification, and you will learn how to check for one. You will finish by building a guardrailed tutor on one of your own lessons and writing an evaluation plan that measures dosage and unassisted learning, not chat volume.

What you will be able to do

1. Convert a reported effect (standard deviations, percentage points, percent change) into a plain-language statement, and compare it with the pooled human-tutoring benchmark of 0.37 standard deviations from the NBER meta-analysis.
2. Appraise any AI tutoring study on a one-page card that records setting, sample size, duration, comparison condition, outcome measure and whether the outcome was assisted or unassisted, and state which of those limit generalization to their institution.
3. Explain the difference between practice-score gains and unassisted-exam outcomes using the PNAS guardrails trial, and specify the guardrails a tutor needs before it reaches students.
4. State plainly what the higher-education evidence base does and does not contain, including that the Harvard two-lesson physics study is the strongest higher-education RCT, and locate any newer synthesis.
5. Build a guardrailed tutor agent on a lesson of their own, test it against a sample student transcript, and document how it gives hints rather than answers.
6. Write a pre-registered evaluation plan for a tutoring pilot with randomization, an unassisted outcome measure, a dosage log and a stopping rule.

Who this course is for

Roles: directors of centers for teaching and learning, digital-learning and online-learning leads, provosts and associate provosts for academic affairs, department chairs evaluating a tutoring pilot, instructional designers, institutional research staff asked to evaluate one, and procurement staff reading vendor evidence claims.

Institutions: any college or university considering, piloting or already running an AI tutor, plus K-12 and workforce programs that want to read the evidence base correctly. The evidence itself is mostly K-12 and international, and the course says so.

Before you start: comfort reading a research abstract. The course teaches effect sizes from scratch. No coding is required; the lab uses an AI agent through a browser on the learner's own lesson materials or a supplied sample lesson.

Format

• Estimated time: 10 to 12 hours.
• Level: intermediate. Assumes no statistics beyond what section 1 teaches.
• Hands-on: one two-part lab in which learners build and stress-test a guardrailed tutor on their own lesson (or a supplied introductory-mechanics lesson), four templates they keep (evidence-appraisal card, dosage plan, tutor specification, evaluation pre-registration), three discussions and a capstone on a real pilot decision.
• Structure: 4 sections, each ending in an AI conversation assessment, then a capstone assessment.

Self-paced. Every lesson is taught by an AI tutor that opens with a short story and works through one idea and one question at a time. Each section ends with an AI conversation assessment, and the course closes with a capstone assessment. Section assessments count for 60% and the capstone for 40%; 70% passes. There are no multiple-choice quizzes.

Currency of this course

Every source in this course was checked on 2026-09-28. Rules and guidance change; check the linked sources before relying on a date or figure.