AI in medical devices · Medium · 13 min lesson
Explainability, automation bias and monitoring of human-AI interaction
Regulation (EU) 2024/1689 Articles 12 to 15, 26 and 72, MDR Articles 83 to 87, ISO 13485:2016 clauses 8.2, 8.4 and 8.5.2, ISO 14971 clause 10, GMLP principles 7 and 10
What you will learn
- Describe an explanation feature honestly, verify it, and state its limits in the instructions for use.
- Recognise errors of commission and omission, explain how automation bias arises, and choose design measures that reduce it.
- Define indicators, baselines and trigger levels for the human and AI team, feed signals into post-market, risk and change control, recognise a weak record and answer the auditor's questions.
Key takeaways
- An explanation feature is a claim: verify it, state its limits and never present a highlight as proof (Article 13 paragraph 3).
- Automation bias has two forms, commission and omission; design to keep the reader's own review active (Article 14 paragraph 4 point b).
- Monitor the human and AI team with defined indicators, baselines and trigger levels set before release (Article 72).
- A falling override rate with faster reading is a signal to investigate, not proof of quality (ISO 13485 clauses 8.4, 8.5.2).
- Feed results into the post-market plan, risk file, change control, instructions and training, and keep the records.
Free practice question
Pneumo-Flag draws a heat map on each study. The instructions say: The highlighted area shows the abnormality. No test of the heat map was carried out.
What is the main problem?
The statement claims more than was verified; the feature needs a tested, honest description under Article 13 paragraph 3 point b.
Correct. Article 13 paragraph 3 point b sub-point iv expects the explanation ability to be described accurately.
There is no problem because heat maps are widely used.
Not quite. Common use is not verification, and clause 7.3.6 of ISO 13485 requires evidence for each design output.
The map must be removed from all software.
Not quite. Nothing in Article 13 forbids explanation features; the requirement is accurate description and verification.
Only the deployer must verify the map.
Not quite. Verification of the product is the provider's responsibility under clause 7.3.6 and Article 13.
Key terms
- explainability: The technical ability of a system to give information that helps users understand why it produced an output (Article 13 paragraph 3 point b).
- automation bias: The tendency to rely too much on an automated output, especially a recommendation used for a human decision (Article 14 paragraph 4 point b).
- error of commission: A human error of following a wrong suggestion from an automated system.
- error of omission: A human error of missing a problem because the automated system gave no alert.
- feedback loop: A situation in which a system's outputs influence the data later used to train or update it (Article 15 paragraph 4).
- post-market monitoring: Active and systematic collection and analysis of performance data over the lifetime of the system (Article 72; MDR Article 83).
Official sources
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Official Journal
- Article 13: Transparency and provision of information to deployers, AI Act explorer
- Article 14: Human oversight, AI Act explorer
- Article 15: Accuracy, robustness and cybersecurity, AI Act explorer
- Article 26: Obligations of deployers of high-risk AI systems, AI Act explorer
Go further
The full lesson has narrated slides, more scenario questions with explained answers and a simulation.
Open the AcademyWatch on YouTubeIllustrative example. Educational only, not legal advice.