AI in medical devices · Expert · 13 min lesson
Cybersecurity for AI devices: threat modelling, adversarial attacks and secure design
MDR Annex I sections 17.2 and 17.4, EU AI Act Article 15, FD&C Act section 524B, NIST AI 100-2 E2025
What you will learn
- Build a threat model for an AI enabled device that covers the data pipeline, the model and the deployed system, with trust boundaries and named assets.
- Distinguish evasion, poisoning, confidentiality and model theft attacks, and link each to Article 15(5) of the EU AI Act and to patient harm.
- Choose layered secure design controls and show in a record how security risk links to safety risk and to design verification.
Key takeaways
- AI devices keep every classic software threat and add attacks on training data, the model and its outputs.
- A threat model names assets, trust boundaries, data flows and threats, and it must cover the training pipeline as well as the deployed product.
- Article 15(5) of the AI Act lists data poisoning, model poisoning, adversarial examples, confidentiality attacks and model flaws for high risk AI.
- Accuracy on a clean test set says nothing about robustness; adversarial and abuse testing are separate verification activities.
- MDR Annex I sections 17.2 and 17.4 and FD&C Act section 524B expect secure development, defined security requirements and documented risk management.
Free practice question
Veridian's threat model TM-12 covers the cloud inference service and its hospital network link. The training data lake, the outside labelling vendor and the model store are not shown. The team says these are internal research tools, not the product.
What is the correct assessment?
Acceptable, because only the deployed service reaches patients.
Not quite. The deployed behaviour is decided upstream, so poisoning there reaches patients; Article 15(5) of the AI Act covers it.
Acceptable if the model has high accuracy.
Not quite. Accuracy does not show that data or model were protected against tampering, which Article 15(5) addresses separately.
The scope is incomplete: training data, labels and model storage are attack routes that change product behaviour.
Correct. Article 15(5) of the AI Act names data and model poisoning, and MDR Annex I section 17.2 expects security across development.
Wrong only because the diagram lacks a legend.
Not quite. A legend is cosmetic; the defect is missing assets and trust boundaries under MDR Annex I section 17.2.
Key terms
- threat model: A structured description of assets, data flows, trust boundaries and the ways an attacker could harm the system, with mitigations.
- trust boundary: A line in the system where data or control passes between parts that deserve different levels of trust.
- adversarial example: An input altered, often invisibly, so that a model gives a wrong output at inference time.
- data poisoning: Manipulation of training data so that the trained model behaves wrongly, sometimes only on a hidden trigger.
- model extraction: Rebuilding a copy of a model by sending many queries and studying the answers.
- membership inference: A confidentiality attack that tests whether a given record was in the training data.
- STRIDE: A threat classification: spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege.
Official sources
- Regulation (EU) 2024/1689, Artificial Intelligence Act, Article 15 (accuracy, robustness and cybersecurity)
- FDA, Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions (guidance page)
- FD&C Act section 524B, 21 U.S.C. 360n-2, Ensuring cybersecurity of devices
- MDCG 2019-16 rev.1, Guidance on cybersecurity for medical devices (July 2020)
- NIST AI 100-2 E2025, Adversarial Machine Learning: a taxonomy and terminology of attacks and mitigations
Go further
The full lesson has narrated slides, more scenario questions with explained answers and a simulation.
Open the AcademyWatch on YouTubeIllustrative example. Educational only, not legal advice.