# Protecting Models and Training Data: Extraction, Membership Inference and Defenses

> Cybersecurity · ibl.ai
> Source: https://ibl.ai/courses/cybersecurity/protecting-models-and-training-data
> Last updated: 2026-11-14

*Make model extraction expensive and visible, protect training data with differential privacy, and lock down weights, with hands-on labs on your own laptop*

**A cybersecurity course from ibl.ai (self-paced, 10 to 12 hours, free).** A hands-on, self-paced course for security, platform and ML teams that serve models through an API: what extraction, membership inference and weights theft achieve and the evidence they leave, then gateway quotas, detection of systematic querying, differential privacy in training and signed artefact checks, taught from OWASP, NIST and MITRE ATLAS.

- **Price:** Free
- **Format:** Self-paced, online
- **Time:** 10 to 12 hours
- **Pace:** 3 hours per week
- **Level:** Intermediate
- **Language:** English
- **Subject:** Information Technology

[Enroll](https://lms.ibl.ai/platform/cybersecurity/courses/course-v1:cybersecurity+protecting-models-and-training-data+self-paced)

## Course description

Protecting Models and Training Data teaches security engineers, AppSec and platform teams, ML engineers and security leaders to protect a served model's weights, behaviour, training data and budget. You will learn the attack classes at a defender's level from OWASP LLM10:2025 Unbounded Consumption and the OWASP AI Exchange, NIST AI 100-2e2025, NIST AI 600-1 and MITRE ATLAS 2026.10: model extraction and behaviour replication, membership and property inference, inversion, extraction of memorised training data, and theft of weights and artefacts. In six local labs you build a gateway in front of a stub model that enforces per-key quotas and refuses risky parameters, verify it with an authorised self-test, detect systematic querying in a synthetic traffic log and tune the thresholds against false positives, train a classifier with DP-SGD in Opacus and read the privacy budget, and verify model files with a signed manifest. The course ends with a model protection plan for a model or fine-tune your organisation serves. Last verified 2026-11-03.

### About this course

A model you serve carries value in four places: its weights and artefacts, the behaviour it exposes through an API, the data it was trained or fine-tuned on, and the bill for every query. This course teaches security, platform and ML teams to protect all four. You will learn the attacks at the level a defender needs (what they achieve, what they require and the evidence they leave) from the primary sources: [OWASP LLM10:2025 Unbounded Consumption](https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/) and the OWASP AI Exchange, [NIST AI 100-2e2025](https://csrc.nist.gov/pubs/ai/100/2/e2025/final), [NIST AI 600-1](https://doi.org/10.6028/NIST.AI.600-1), [NIST SP 800-226](https://csrc.nist.gov/pubs/sp/800/226/final), [MITRE ATLAS](https://github.com/mitre-atlas/atlas-data) and the joint guidance [Deploying AI Systems Securely](https://www.cisa.gov/news-events/alerts/2024/04/15/joint-guidance-deploying-ai-systems-securely). Then you build and test the controls on your own laptop: a gateway with per-key quotas in front of a stub model, a detector for systematic querying tuned against false positives, a differentially private training run on synthetic data, and a signed integrity check for model files. Every lab is authorised self-testing on local systems with synthetic data. The principles are vendor-neutral; where a control exists on ibl.ai, the course links the live documentation.

### What you will be able to do

- Name the four assets a served model exposes (weights and artefacts, behaviour, training data and the bill) and explain model extraction, behaviour replication, membership and property inference, inversion and memorised-data extraction at a defender's level: what the attacker achieves, the preconditions and the evidence, mapped to OWASP LLM10:2025, NIST AI 100-2e2025 and MITRE ATLAS 2026.10.
- Design front-door controls for a model API: authentication and scoped keys, tiered rate limits and quotas including aggregate limits per tenant, workload and spend limits, and output minimisation for probabilities, parameters and reasoning traces.
- Build a local API gateway in front of a stub model that enforces per-key quotas, refuses risky parameters and logs key fingerprints, and verify every control with an authorised self-test.
- Detect systematic querying in gateway logs from volume, prompt similarity, deterministic settings, round-the-clock timing, new-key ramps and templates shared across keys, tune thresholds against false positives, and respond in proportion with evidence preserved.
- Explain what differential privacy in training protects and what it does not, choose and record epsilon, delta and the unit of privacy, run DP-SGD with Opacus on synthetic data, and state the published limits of output perturbation and watermarking.
- Protect weights and artefacts with least-privilege access, isolation, encryption and a signed integrity manifest, and write a model protection plan with tests, owners, accepted residual risk and review triggers.

### Who this course is for

**Roles:** security engineers, AppSec and platform teams, ML engineers and security leaders who serve a model, a fine-tune or an AI feature through an API, in any sector.

**Organisations:** companies, public bodies and institutions that train, fine-tune or host models for customers, partners or staff.

**Before you start:** comfort with HTTP APIs, logging and identity and access management, and enough Python to run and edit short scripts. The labs need Python 3; Lab 5 also installs Opacus 1.6.0 and PyTorch 2.15.0 in a separate environment. All lab data is synthetic and supplied in the course.

### Format

**Time:** about 10 to 12 hours, self-paced. **Level:** intermediate to advanced. **Hands-on:** six local labs in Python, with an AI tutor that coaches each step from the output you paste.

**Structure:** a Start here section, four content sections of readings, worked examples, templates and hands-on defensive labs, each ending in an AI assessment, and a capstone in which you write a model protection plan for a model or fine-tune your organisation serves, followed by a reading list.

**Grading:** the four section AI assessments count for 60 percent of the grade and the capstone assessment for 40 percent. The pass mark is 70 percent.

### Currency

**Last verified 2026-11-03** against the OWASP Top 10 for LLM Applications 2025, the OWASP AI Exchange, NIST AI 100-2e2025, NIST AI 600-1, NIST SP 800-226, NIST AI 100-4, NIST SP 800-61 Rev. 3, MITRE ATLAS release 2026.10, the joint guidance Deploying AI Systems Securely, joint advisory AA26-251A, Opacus 1.6.0 and the ibl.ai documentation.

This field changes quickly. ATLAS release 2026.10 moved the membership and inversion techniques to new identifiers, OWASP and NIST revise their guidance often, and libraries change between versions. Check the current versions before relying on any page or template, and test only systems you own or are authorised to test. This course is training, not a security assessment of your systems.

## Outline

### Section 0 · Start here

- 0.1 Welcome and orientation

### Section 1 · Know what is at risk: extraction, training data and weights

- 1.1 The assets and the attack classes
- 1.2 Theft by other routes, and a first map
- 1.3 Section 1 assessment

### Section 2 · Control the front door: identity, quotas and output minimisation

- 2.1 Gateway controls
- 2.2 Build and verify a gateway
- 2.3 Section 2 assessment

### Section 3 · Detect systematic querying and respond

- 3.1 Signals and detection
- 3.2 Respond without breaking legitimate use
- 3.3 Section 3 assessment

### Section 4 · Protect the training data and the weights

- 4.1 Training-time and output defences
- 4.2 Weights, artefacts and the plan
- 4.3 Section 4 assessment

### Section 5 · Capstone and wrap-up

- 5.1 Capstone: a model protection plan for a model you serve
- 5.2 Capstone assessment
- 5.3 Reading list and next steps

## Instructors

- ibl.ai Cybersecurity, Course team (ibl.ai)

## More cybersecurity courses

- [Protecting PII in LLM Applications: Masking, Tokenization and Privacy Proxies](https://ibl.ai/courses/cybersecurity/protecting-pii-in-llm-applications)
- [Securing RAG: Access Control, Provenance and Poisoning Defenses](https://ibl.ai/courses/cybersecurity/securing-rag-pipelines)
- [Continuous Security Testing for LLM Apps: Automated Evaluation and CI Gates](https://ibl.ai/courses/cybersecurity/continuous-security-testing-for-llm-apps)
- [Coordinated Vulnerability Disclosure for AI Systems](https://ibl.ai/courses/cybersecurity/ai-vulnerability-disclosure)