📅 Book a 30-min Demo📞 Call/text (571) 293-0242

Interpretability for AI Assurance: Inspecting Model Internals

Cache activations, read the logit lens, patch and probe a model on your laptop, and decide where interpretability evidence can and cannot support a deployment decision

A cybersecurity course from ibl.ai (self-paced, 10 to 12 hours, free). A hands-on, self-paced course for security and ML teams on mechanistic interpretability as assurance evidence. Learn why black-box testing leaves gaps, run activation caching, the logit lens, activation patching and probing on GPT-2 small with TransformerLens, read what research found on sycophancy, reward hacking and hidden objectives, and write an assurance plan that states what interpretability can and cannot certify.

Last updated:

Course description

Interpretability for AI Assurance teaches security engineers, AppSec and platform teams, ML engineers and security leaders to use evidence from inside a model alongside behavioural testing. You will learn why query-only testing leaves gaps and which behaviours exploit them (sycophancy, reward hacking, backdoors and hidden objectives), what NIST AI 600-1, NIST AI 100-2e2025, OWASP, MITRE ATLAS and CISA guidance say about interpretability, and which access level your model allows. In four labs on GPT-2 small with TransformerLens 4.3, run on a laptop CPU, you cache activations, apply the logit lens, run an activation-patching experiment and train a linear probe with a control, recording each result in a finding record. You then read the published research on sycophancy, reward hacking, hidden-objective audits and probe-based monitors, and learn what interpretability can and cannot certify today. The course ends with an assurance plan for a model you use. Last verified 2026-11-03.

About this course

Most AI assurance is behavioural: you test a model with prompts and judge the outputs. Research has shown behaviours that can pass such tests, from sycophancy to backdoors that survive safety training. Mechanistic interpretability offers a second source of evidence by looking inside the model. This course teaches its core methods from the primary papers and the TransformerLens documentation: activation caching, the logit lens, activation patching and circuit discovery, probing and sparse autoencoders. You run them on GPT-2 small on your own laptop, read what researchers found about sycophancy, reward hacking and hidden objectives, and then decide, as a defender, where this evidence can and cannot support a deployment decision. The course is grounded in NIST AI 600-1, NIST AI 100-2e2025, the OWASP Top 10 for LLM Applications 2025, MITRE ATLAS and the International AI Safety Report 2026, which states plainly that interpretability cannot yet provide robust, quantifiable assurances. You will learn to use it at exactly that strength.

What you will be able to do

  1. Explain why black-box testing alone leaves assurance gaps, and describe how sycophancy, reward hacking, backdoors and hidden objectives can pass behavioural tests, with their preconditions and the evidence they leave.
  2. Cite accurately what NIST AI 100-1, NIST AI 600-1, NIST AI 100-2e2025, OWASP, MITRE ATLAS and CISA guidance say about interpretability, and state which access level a given model allows.
  3. Set up TransformerLens 4.3 with GPT-2 small on a laptop CPU, cache activations and apply the logit lens, and report what the results show without over-claiming.
  4. Design and run an activation-patching experiment with matched prompts and a logit-difference metric, and train a linear probe with held-out data and a control.
  5. Assess published interpretability findings on sycophancy, reward hacking and hidden objectives, and design a probe-based monitor with measured detection and false-positive rates.
  6. Write an interpretability assurance plan that states where interpretability evidence would and would not support a deployment decision for a model you use.

Who this course is for

Roles: security engineers, AppSec and platform teams, ML engineers, AI red teams and security leaders who build, buy or run LLM applications and agents and must decide what evidence supports deploying a model.

Organisations: any sector using language models, whether hosted through an API or self-hosted from open weights.

Before you start: familiarity with LLM applications and with security testing. For the four labs you need a laptop with Python 3.12, about 1 GB of disk and comfort running Python scripts; no GPU is needed. Every lab also shows the output of a tested run, so you can follow it without coding. All lab prompts are synthetic, and every test is authorised self-testing on your own copy of a public open-weights model.

Format

Time: about 10 to 12 hours, self-paced. Level: intermediate to advanced. Hands-on: four labs on GPT-2 small with TransformerLens 4.3, with an AI tutor that coaches each step from the output you paste.

Structure: a Start here section, four content sections of readings, worked examples, templates and hands-on labs, each ending in an AI assessment, and a capstone in which you write an interpretability assurance plan for a model you use, followed by a reading list.

Grading: the four section AI assessments count for 60 percent of the grade and the capstone assessment for 40 percent. The pass mark is 70 percent.

Currency

Last verified 2026-11-03 against TransformerLens 4.3.0, the primary papers cited in each unit, NIST AI 100-1, NIST AI 600-1, NIST AI 100-2e2025, the OWASP Top 10 for LLM Applications 2025, OWASP Agentic AI: Threats and Mitigations (version 1.1, December 2025), the OWASP AI Testing Guide (version 1), MITRE ATLAS release 2026.10, the International AI Safety Report 2026 and the ibl.ai documentation.

This field changes quickly. Interpretability methods and tools are revised often (TransformerLens 4.0 removed the API most older tutorials use), and OWASP, NIST and MITRE update their guidance between releases. Check current versions before relying on any page or template, and inspect or test only models and systems you are licensed and authorised to use. This course is training, not an assessment of any model.

Outline

  1. Section 0 · Start here

    • 0.1 Welcome and orientation

  2. Section 1 · Find the gaps black-box testing leaves

    • 1.1 Why behavioural tests are not enough
    • 1.2 Where interpretability sits in the guidance
    • 1.3 Section 1 assessment

  3. Section 2 · Read a model's internals: caching and the logit lens

    • 2.1 The residual stream and your lab
    • 2.2 Reading predictions layer by layer
    • 2.3 Section 2 assessment

  4. Section 3 · Test causal claims: patching, circuits and probes

    • 3.1 Activation patching and circuits
    • 3.2 Probes, features and how to record a finding
    • 3.3 Section 3 assessment

  5. Section 4 · Use interpretability evidence in an assurance decision

    • 4.1 What research shows on sycophancy, reward hacking and hidden objectives
    • 4.2 What interpretability can and cannot certify
    • 4.3 Section 4 assessment

  6. Section 5 · Capstone and wrap-up

    • 5.1 Capstone: an assurance plan for a model you use
    • 5.2 Capstone assessment
    • 5.3 Reading list and next steps