AI in medical devices · Medium · 13 min lesson
Bias, drift, misuse and human factors risk controls with worked examples
ISO 14971:2019 clauses 5, 7 and 10, IEC 62366-1, EU AI Act Articles 9, 10, 14 and 15, with a fictional wrist rhythm app
What you will learn
- Examine a data set and a model for bias with predefined subgroups, confidence limits and a predefined limit, and select controls in the order of ISO 14971 clause 7.1.
- Design drift monitoring with a metric, a threshold, an owner and an action, and tie model updates to change control.
- Analyse reasonably foreseeable misuse and automation bias, and show usability and field evidence an auditor can follow.
Key takeaways
- Predefine the groups and the acceptance limit, then report subgroup results with confidence limits (AI Act Article 10(2)(f) and (g)).
- Bias controls start with design, such as signal quality gating, and are verified by repeating the subgroup test.
- Drift monitoring needs a metric, threshold, owner and action; leading indicators warn before confirmed harm accumulates (clause 10).
- Retraining is a device change: risk assessment, regression on subgroup limits and approval under change control.
- Foreseeable misuse (clause 5.2) and automation bias (Article 14(4)(b)) are controlled by design and usability evidence under IEC 62366-1.
Free practice question
PulseGuard shows 93 percent sensitivity overall. The darkest skin tone band shows 84 percent with a lower limit of 79. The plan allowed a drop of at most five points below overall.
What is the correct conclusion?
The group fails the predefined limit, so the hazard is estimated with the group result and a control is needed.
Correct. Article 10(2)(f) requires bias examination and ISO 14971 clause 5.5 estimation must reflect the exposed group.
The device passes because the overall figure is above 90 percent.
Not quite. The overall figure hides the weak group, contrary to the examination of bias in Article 10(2)(f).
The result is acceptable because the group is small.
Not quite. Small size increases uncertainty, so clause 5.5 calls for caution and not for dismissal.
The limit should be raised after seeing the result.
Not quite. Changing the limit after the result undermines the predefined plan expected under Article 10.
Key terms
- bias: Systematic difference in performance or outcome across groups, conditions or devices that can affect safety.
- data drift: A change in the input data after release, for example from a new sensor firmware, that can lower model performance.
- concept drift: A change in the relationship between inputs and the true outcome, for example after a clinical guideline changes.
- reasonably foreseeable misuse: Use in a way not intended by the manufacturer that can result from readily predictable human behaviour (ISO 14971 clause 5.2).
- automation bias: Over-reliance on an automated output (AI Act Article 14(4)(b)).
- summative evaluation: Usability testing with representative users that shows the device can be used safely (IEC 62366-1).
- signal quality index: A measure of whether an input signal is good enough for the model to give a reliable result.
Official sources
- ISO 14971:2019 Medical devices - Application of risk management to medical devices
- IEC 62366-1:2015 Medical devices - Application of usability engineering to medical devices (ISO catalogue)
- Article 9: Risk management system, AI Act explorer
- Article 10: Data and data governance, AI Act explorer
- Article 14: Human oversight, AI Act explorer
Go further
The full lesson has narrated slides, more scenario questions with explained answers and a simulation.
Open the AcademyWatch on YouTubeIllustrative example. Educational only, not legal advice.