AI in medical devices · Medium · 13 min lesson
Clinical validation, subgroup performance, robustness and reporting of results
MDCG 2020-1, MDR Article 61, Regulation (EU) 2024/1689 Articles 10, 15, GMLP principles 3 to 9, IEC 62366-1
What you will learn
- Explain the three elements of clinical evidence for software under MDCG 2020-1 and design a clinical validation around the intended use.
- Plan and interpret subgroup, bias and robustness analyses, including small groups with wide intervals.
- Write a results report and recognise a weak one, and answer the questions an auditor asks about it.
Key takeaways
- MDCG 2020-1: valid clinical association, technical performance and clinical performance are all needed for software.
- Choose subgroups from the risk analysis before testing, and report each with counts and intervals.
- A failed subgroup is a finding: report it, investigate, and change the model, claim or labelling, then re-verify.
- Robustness and security testing support Article 15 and the risk file, including out-of-scope inputs.
- Reports state population, reference standard, intervals, prevalence, limitations and actions; the instructions for use declare performance (Article 15 paragraph 3).
Free practice question
Corvane's retrospective study reports sensitivity 0.89 overall. A reviewer finds that 85 percent of images came from one hospital with fixed scanners, and the intended use covers all hospital settings including intensive care with portable units.
What is the main concern?
The data may not be representative of the intended population, so subgroup and site results are needed.
Correct. Article 10 paragraph 3 and GMLP principle 3 require data that reflect the intended patients, settings and acquisition conditions.
None, because 0.89 is high enough.
Not quite. A high average does not show representativeness; Article 10 paragraph 3 asks for suitable properties for the groups affected.
The overall sample is too large.
Not quite. Size is not the concern; the concentration in one site and scanner type limits what the evidence supports under Article 10 paragraph 3.
Only the user interface needs change.
Not quite. The issue is evidence coverage, which GMLP principle 8 addresses through realistic conditions and subgroups.
Key terms
- clinical validation: Evidence that the output achieves the intended clinical purpose in the target population and with the intended users.
- reference standard: The best available method used to establish the true condition against which the device output is compared.
- subgroup analysis: Reporting performance separately for pre-specified groups, such as age, sex, site or scanner.
- robustness: The ability to keep performing acceptably when inputs or conditions deviate from the ideal.
- out-of-distribution input: An input unlike the data the model was built for, which should be detected or rejected.
- automation bias: The tendency of users to over-trust automated output and reduce their own checking.
- reader study: A study in which clinicians interpret cases with and without the device to measure the team effect.
- TRIPOD+AI: A 2024 reporting guideline for studies of prediction models that use regression or machine learning.
Official sources
- MDCG 2020-1 Guidance on clinical evaluation (MDR) / performance evaluation (IVDR) of medical device software
- Article 15: Accuracy, robustness and cybersecurity, AI Act explorer
- Article 10: Data and data governance, AI Act explorer
- Good Machine Learning Practice for Medical Device Development: Guiding Principles (MHRA, with FDA and Health Canada)
- Regulation (EU) 2017/745 (Medical Device Regulation), Official Journal
Go further
The full lesson has narrated slides, more scenario questions with explained answers and a simulation.
Open the AcademyWatch on YouTubeIllustrative example. Educational only, not legal advice.