# AI Tutoring Research: What the RCTs Show and How to Deploy a Tutor That Improves Learning > Higher Education · ibl.ai > Source: https://ibl.ai/courses/higher-education/ai-tutoring-research-what-rcts-show > Last updated: 2026-11-14 *What the RCTs Show and How to Deploy a Tutor That Improves Learning* **A higher education course from ibl.ai (self-paced, 10 to 12 hours, free).** Read the AI tutoring RCTs properly: Harvard's two-lesson physics study, the PNAS guardrails trial, human-AI hybrids, dosage. Then run your own evaluation. - **Price:** Free - **Format:** Self-paced, online - **Time:** 10 to 12 hours - **Pace:** 3 hours per week - **Level:** Intermediate - **Language:** English - **Subject:** Teaching & Learning [Enroll](https://lms.ibl.ai/platform/highereducation/courses/course-v1:highereducation+ai-tutoring-research-what-rcts-show+self-paced) ## Course description AI tutoring research is now large enough to mislead: a decision-maker can find a randomized trial to support almost any position, from "outperforms active learning" to "harms exam scores." This course teaches you to read the trials as a body of evidence, benchmark them against human tutoring, and design a deployment that follows what the studies actually found. You will learn why unguarded GPT-4 raised practice scores by 48 percent and cut unassisted exam scores by 17 percent in the PNAS trial (2025), why the Harvard physics study's 0.73 to 1.3 standard-deviation effect covers two lessons in one course with expert-written prompts, and why the strongest higher-education trial to date is exactly that two-lesson Harvard study, with the rest of the rigorous evidence coming from K-12 and international settings. No higher-education meta-analysis of generative-AI tutoring had been located at verification, and you will learn how to check for one. You will finish by building a guardrailed tutor on one of your own lessons and writing an evaluation plan that measures dosage and unassisted learning, not chat volume. ### About this course AI tutoring research is now large enough to mislead: a decision-maker can find a randomized trial to support almost any position, from "outperforms active learning" to "harms exam scores." This course teaches you to read the trials as a body of evidence, benchmark them against human tutoring, and design a deployment that follows what the studies actually found. You will learn why unguarded GPT-4 raised practice scores by 48 percent and cut unassisted exam scores by 17 percent in the PNAS trial (2025), why the Harvard physics study's 0.73 to 1.3 standard-deviation effect covers two lessons in one course with expert-written prompts, and why the strongest higher-education trial to date is exactly that two-lesson Harvard study, with the rest of the rigorous evidence coming from K-12 and international settings. No higher-education meta-analysis of generative-AI tutoring had been located at verification, and you will learn how to check for one. You will finish by building a guardrailed tutor on one of your own lessons and writing an evaluation plan that measures dosage and unassisted learning, not chat volume. ### What you will be able to do - Convert a reported effect (standard deviations, percentage points, percent change) into a plain-language statement, and compare it with the pooled human-tutoring benchmark of 0.37 standard deviations from the NBER meta-analysis. - Appraise any AI tutoring study on a one-page card that records setting, sample size, duration, comparison condition, outcome measure and whether the outcome was assisted or unassisted, and state which of those limit generalization to their institution. - Explain the difference between practice-score gains and unassisted-exam outcomes using the PNAS guardrails trial, and specify the guardrails a tutor needs before it reaches students. - State plainly what the higher-education evidence base does and does not contain, including that the Harvard two-lesson physics study is the strongest higher-education RCT, and locate any newer synthesis. - Build a guardrailed tutor agent on a lesson of their own, test it against a sample student transcript, and document how it gives hints rather than answers. - Write a pre-registered evaluation plan for a tutoring pilot with randomization, an unassisted outcome measure, a dosage log and a stopping rule. ### Who this course is for **Roles:** directors of centers for teaching and learning, digital-learning and online-learning leads, provosts and associate provosts for academic affairs, department chairs evaluating a tutoring pilot, instructional designers, institutional research staff asked to evaluate one, and procurement staff reading vendor evidence claims. **Institutions:** any college or university considering, piloting or already running an AI tutor, plus K-12 and workforce programs that want to read the evidence base correctly. The evidence itself is mostly K-12 and international, and the course says so. **Before you start:** comfort reading a research abstract. The course teaches effect sizes from scratch. No coding is required; the lab uses an AI agent through a browser on the learner's own lesson materials or a supplied sample lesson. ### Format - Estimated time: 10 to 12 hours. - Level: intermediate. Assumes no statistics beyond what section 1 teaches. - Hands-on: one two-part lab in which learners build and stress-test a guardrailed tutor on their own lesson (or a supplied introductory-mechanics lesson), four templates they keep (evidence-appraisal card, dosage plan, tutor specification, evaluation pre-registration), three discussions and a capstone on a real pilot decision. - Structure: 4 sections, each ending in an AI conversation assessment, then a capstone assessment. Self-paced. Every lesson is taught by an AI tutor that opens with a short story and works through one idea and one question at a time. Each section ends with an AI conversation assessment, and the course closes with a capstone assessment. Section assessments count for 60% and the capstone for 40%; 70% passes. There are no multiple-choice quizzes. ### Currency of this course Every source in this course was checked on 2026-09-28. Rules and guidance change; check the linked sources before relying on a date or figure. ## Outline ### Section 0 · Start here - 0.1 Orientation ### Section 1 · Learn to read a tutoring study - 1.1 Effect sizes and the human benchmark - 1.2 The evidence map, and the higher-education gap - 1.3 Section 1 assessment ### Section 2 · Know the studies decision-makers cite - 2.1 Harvard: the headline and its caveats - 2.2 Guardrails: the PNAS trial - 2.3 Scale and hybrids: Ghana, Nigeria, Tutor CoPilot and LearnLM - 2.4 Section 2 assessment ### Section 3 · Understand why deployments fail - 3.1 Dosage: the silent failure - 3.2 Dependence and accuracy - 3.3 Section 3 assessment ### Section 4 · Design and deploy a tutor that follows the evidence - 4.1 Building the tutor - 4.2 Running your own evaluation - 4.3 Section 4 assessment ### Section 5 · Capstone & wrap-up - 5.1 Capstone - 5.2 Capstone assessment - 5.3 Wrap-up ## Instructors - ibl.ai Higher Education, Course team (ibl.ai) ## More higher education courses - [Assessment Redesign for Generative AI: Program-Level Assurance of Learning](https://ibl.ai/courses/higher-education/assessment-redesign-for-generative-ai) - [Designing a Faculty AI Training Program: Frameworks, Tiers and Evaluation](https://ibl.ai/courses/higher-education/faculty-ai-training-program-design) - [ADA Title II Digital Accessibility for Course Content: The 2027 Deadline](https://ibl.ai/courses/higher-education/ada-title-ii-course-accessibility) - [AI Academic Advising at Scale: Caseload Triage, Proactive Outreach and Human Handoffs](https://ibl.ai/courses/higher-education/ai-academic-advising-caseload-management)