How To Obtain CE Approval for Medical Diagnostic Models

When it comes to medical AI, obtaining regulatory approval to sell a product within the EU is a long and complicated process. This article provides an overview of the steps required to obtain CE approval under the EU MDR, including working with a Notified Body, developing a QMS that aligns with best practices for ISO 13485 certification, establishing the Intended Use and device classification for medical technology, and compiling the documents required for the Technical Files audit for CE approval. 

‍Medical devices date back to early civilization. In 6000 BCE, Neolithic groups fashioned knives, saws, and drills out of stones for use in surgery, amputation, and dental work. Historical records show that by 500 CE, the Greeks and Romans had documented the widespread use of medical instruments. And in the mid-1800s, scientific advancement led to a proliferation of medical devices being used to treat soldiers’ wounds and the ailments of wealthy families.  

Regulating the use of medical devices, however, is a much newer concept, which largely came about after World War II when an explosion of technological progress and manufacturing capacity resulted in a surge of devices being sold to hospitals and doctors. In the years since, when incidents involving medical devices have occurred, governments have reacted accordingly, changing existing regulations and implementing new ones.

At their core, these regulations– which vary from country to country–exist to protect patients. While regulators have traditionally focused on physical products (such as scalpels, thermometers, and prosthetics), the acceleration of technology and expansion of engineering has resulted in the increased use of AI and machine learning within the medical field. 

When used for medical purposes, machine learning models and the algorithms used to build them are considered medical devices and, for regulatory approval purposes, often considered components of healthcare software. 

Unfortunately, technology evolves much more quickly than regulatory guidelines. Often, the medical device guidelines are somewhat generic, designed for generalisation and not for the nuances of building deep learning algorithms. As a result, AI companies may experience tension between complying with regulations and developing medical machine learning models designed for continuous improvement. 

Collaborative DICOM annotation platform for medical imaging

CT, X-ray, mammography, MRI, PET scans, ultrasound

See it in action

Nonetheless, meeting the essential requirements of the regulations for healthcare software in your intended market is paramount for commercializing your medical diagnostic model.

Below is a high-level overview of the current (2022) process for meeting regulatory approval to deploy a medical model in the European Union.

‍Understanding European Regulatory Approval: The Difference Between the CE Mark and the MDR 

To sell any product in Europe, a company must obtain a CE marking for that specific good. CE stands for “Conformité Européene,” and the mark indicates that the manufacturer has confirmed the product conforms to European health, safety, and environmental protection standards. Once a product has CE approval, it can be sold throughout the European Economic Area, regardless of where it’s made.

The essential requirements for products vary depending on the product type. Different product types must comply with the standards of different Directives set by the European Commission. Medical devices, including healthcare models, must meet the standards laid out in the EU Medical Device Regulation (MDR). 

To receive a CE marking for a diagnostic model, an AI company must show that its technology complies with the essential requirements outlined in the MDR. However, AI companies should think from a software perspective rather than a model-only perspective. Because a model is part of the overall software, a manufacturer needs to ensure that all components of software intended for diagnostic support work well. If a model predicts correctly, but the software has a bug that causes an incomprehensible output, then the AI company won’t receive CE approval for its product.

To obtain CE approval, the manufacturer must determine the appropriate medical device classification level for their healthcare software and complete the conformity assessment for that classification level.

‍Who Approves the CE Marking for Medical AI?

The European Commission doesn’t actually judge firsthand whether the medical product fulfills the essential requirements of the Directive. Instead, they rely on private, for-profit, external organizations known as Notified Bodies to perform the conformity assessment. 

‍The EU has approved approximately 25 Notified Bodies to carry out these assessments for medical software, and the company can work with whichever one they like. However, the company is also responsible for paying the Notified Body to conduct the audit. 

For all companies, but especially startups with more limited resources, the cost and time required to navigate these complex regulations can create barriers to getting a product to market. Many larger AI companies will opt to work with consultants to ensure that the process is successful on the first attempt. However, since getting a product through regulation alone can cost tens of thousands of dollars, for many startups, hiring external expertise isn’t a viable option. In this case, startups in the medical AI ecosystem should consider working together, sharing information about similar technologies that have received regulatory approval, and imitating the path previously taken.

When building medical AI, companies shouldn’t underestimate the need to think about regulatory approval from the get-go. If you don't document your processes and design your healthcare model with these regulations in mind, you run the risk of accumulating a lot of technical debt or being unable to sell your product to consumers.

‍Obtaining CE Approval for Healthcare AI

The first step in securing a CE mark for your AI is declaring its Intended Use, which is exactly what it sounds like you must declare what your product does, including the medical purpose and product description (e.g., how it works), who should use it, and how they should use it. 

The Intended Use is the foundation of the conformity assessment. It is a lengthy description that provides a detailed level of documentation outlining both the Intended Use and how and why the company concluded that this was the best and most appropriate use of the technology. Within this section, you’ll need to include a lot of information, such as a description of the AI, its purpose and use environment, intended users and patient populations, anticipated risks, and other details.

The Intended Use document describes what the software should do, not what it could do. As you develop and evaluate the technology, you’ll work to prove that it achieves this purpose via software tests, user tests, clinical validation, and more. 

Start-ups operate at a fast pace, and their ability to adapt as they gain new information is often beneficial for product performance. However, for higher-risk devices (Class II and above), the MDR prevents any “significant changes” to the design (such as introducing a new machine learning algorithm) or the product’s Intended Use (such as diagnosing a different disease or treating a different part of the body) because these changes could affect the product’s safety and performance.

When it comes to algorithms and models, there’s still some debate about what qualifies as a “significant” change. The European Commission has provided some guidance, but ultimately, the Notified Body will decide on a case-by-case basis whether a change is significant. The best strategy for AI companies seeking regulatory approval quickly is to make as few changes as possible to the model once you’ve declared your Intended Use. While doing so may hinder technological innovation, it prevents an AI company from encountering the long review timelines and additional costs that often accompany obtaining approval for a significant change.

The Intended Use will function as a roadmap for regulators. After you outline the purpose of your technology and patient population that it will serve, the Notified Body will evaluate whether the technology actually achieves that purpose for that population in a safe and effective manner. That’s why if you attempt to change this description later on, you’ll have to start the entire conformity assessment again.

When establishing the Intended Use, you’ll also need to determine the medical device classification of your software.

The EU ranks devices from Class I to Class III based on the risks they pose to patients. Class I devices, such as bandages and crutches, have low risks to patients and do not include a measuring function. 

Class II devices, which pose moderate risks, are broken down into two subsets. Class II.a. carries less risk, such as blood pressure cuffs and syringes, while Class II.b. poses a slightly higher risk and often has direct contact with the human body, such as anesthesia machines and ventilators. 

Class III are high-risk devices, like pacemakers and stents.

The higher the class of the device, the more thorough the conformity assessment required and the more intense the regulatory scrutiny. Because the consequence of a malfunctioning device rises with the class level, so does the manufacturer’s burden to prove that the device is both safe for patients and beneficial to patients or medical professionals. 

Because medical machine learning models are usually considered a part of healthcare software, they mostly fall under Class IIa or Class IIb. The more autonomous the AI, the more likely it is to fall into the higher class. 

For a Class II Medical Device, the conformity assessment includes a two-part audit by the Notified Body. Part one of the audit requires that a company show that the Quality Management System (QMS) used for its software development complies with International Standards of Organizations regulations for medical devices (ISO 13485) – regulation which is independent of, but connected with, the MDR. Part two requires that the company produce specific technical documentation (known as Technical Files) that details the development and testing of its technology.

‍Compiling the Documentation Required for European Regulatory Approval

To pass the QMS audit, a company must provide the Notified Body with a collection of documents that details the processes and procedures used to develop the healthcare software. Expect to provide hundreds of pages documenting dozens of standard operating procedures.

In general, the documents provided for the QMS must show the operating procedures that the company used when developing its software, including the medical model. You’ll need to show the processes you used to collect, anonymize, label, and store patient data. Keep in mind that your data processes must be compliant with data protection and privacy laws, which means navigating an inherent tension between providing evidence that your healthcare model works for the intended population (a requirement for CE approval) and adhering to GDPR regulations. 

For instance, let’s say that to prove the benefit of your technology; you need to show that your diagnostic model performs exceptionally well on men aged 40 to 60. In doing so, you’ll also have to think about protecting the identity of patients. When you begin collecting patient data, you’ll want to structure the data so that it meets both requirements. Rather than list the patient’s age, consider bucketing the data into age ranges of 10 years to help ensure privacy.

Modern machine learning algorithms must be trained on vast amounts of annotated data, and you’ll need to complete an audit trail about where the data came from, who annotated it, and how. Having the correct technology will make producing this trail much simpler; for instance, having a tool like Encord that natively supports DICOM and logs the history of how annotations were created, including time stamps and sequential recording, will help you keep an audit trail for each individual label.

In addition to achieving an ISO 13485 certification for its QMS, the company must provide the Notified Body with its Technical Files to achieve CE approval under the MDR. The Technical Files is a collection of documents (also hundreds of pages long) designed to show, via real-world evidence, that a medical device meets the necessary safety and performance requirements. It contains all testing reports dating back to the conception of the technology. 

The Technical Files also contains marketing material, such as website copy, product descriptions, and manuals, but the majority of its documents are related to the performance of the healthcare software, including the medical model. You must show that the machine learning model performs its intended use at a certain threshold for the intended population and support these claims with real-world evidence.

Within these technical documents, you’ll also need to show records of how you built your models and performed your testing. As much as possible, you’ll want to be sure these processes and procedures were in place, documented, and followed from the beginning of AI development. Interestingly, the Technical Files don’t actually contain the codebase used to write algorithms, nor does it contain the datasets used to train, validate, and test model performance. It’s a document log that details how the AI was developed rather than a collection of the material that went into making the AI.

You’ll need to include design specifications that detail acceptable functionality as well as the measures taken to ensure the device works as intended. You’ll also need to detail the system architecture, including how this AI interacts with other systems that might be found in its intended use environment. When it comes to devise performance, you’ll have to show that the model achieves its intended purpose by reporting on metrics such as accuracy, sensitivity, specificity, and recall rate. 

The Technical Files also contain the Clinical Evaluation Report. This report details all the performance testing done during research, development, and pre-market release to show that your healthcare software adds additional clinical benefits, such as improving the speed or accuracy of diagnoses. 

To determine and justify the threshold for the performance of the model, you need to include a literature review in which you perform a meta-analysis about equivalent technology and medical practices. In this review, you’ll establish the performance level for state-of-the-art technology and provide context for showing that your technology goes beyond the existing state-of-the-art options and for demonstrating that the medical benefits of doing so outweigh any potential risks of using AI.

You’ll prove these claims with a clinical investigation that reports and explains software results, including the results of the model’s performance. It’s worth noting that the report requires a study containing clinical data but not the conduction of clinical trials. An AI company can partner with research hospitals to conduct trials, but for most companies, a retrospective analysis is faster, cheaper, and more practical. In a retrospective analysis, your company can obtain previously annotated medical data and compare the notes and predictions made by medical professionals against the predictions made by your diagnostic model. 

You should round out the report with a benefit-risk-analysis conclusion– which references any anticipated risks laid out in the Technical Files and Intended Use– and then make a recommendation for using the healthcare software. Taken as a whole, the clinical evaluation report ultimately serves to summarise the technology’s required performance benchmarks, risks, and benefits, clinical performance, compliance with general safety requirements, and limitations.

In addition to the above documentation, the audit process also requires that the head of the company submit a signed Declaration of Conformity in which they assert that all aspects of production and testing conform to EU mandates.

‍The Impact of European Regulatory Approval on Company Employees

Remember that undergoing these audits can be a stressful time for your employees. 

As part of the QMS audit, Notified Body wants to ensure that your company is “living the QMS.” Because standard operating procedures aim to reduce errors, the auditor needs to ensure that a company hasn’t just written these procedures but that their employees actually follow them on the day-to-day. 

The Notified Body will send representatives to the company to ask questions about the information contained in the submitted documents. The company must declare which person is responsible for each task, and the representatives will individually interview these employees, asking questions like “How has this model been handed to production? How has this process been documented? Who had access to the data?”

While the Notified Body officials won’t necessarily require complicated answers, they do expect that your employees will be able to provide commentary on the document trail, including information about who was involved in the product, where the data came from, and where it was stored, the type of tests that were conducted, and various other aspects of the technological development. 

The Notified Body is looking to ensure there are no deviations, known as “nonconformities,” between what is done in practice and how it should be done according to your standard operating procedure. When it comes to the QMS, companies should aim to “do what they document, and document what they do.”

‍The End of the CE Approval Process

Once the Notified Body confirms that the software passes the essential requirements of the Medical Regulation Directive, your company will receive a Declaration of Conformity for the product. You will have to display the CE mark on your product before you can release it on the market. Make sure to follow the guidelines specific to your product type.

Congratulations! You can now market and sell your medical diagnostic model throughout the EU!

Collaborative DICOM annotation platform for medical imaging

CT, X-ray, mammography, MRI, PET scans, ultrasound

See it in action

‍Conclusion: A Quick Summary of the EU MDR CE Approval Process for Healthcare AI

• Develop procedures for data collection and model development that align with QMS best practices. Ensure that other aspects of quality management needed for ISO 13485 certification– employee qualifications, supplier evaluations, and external software tests– are in place. (If possible, obtain validation reports from suppliers to lighten the internal workload required.)
• Consider working with a regulatory approval consultant or connecting with other startups in the medical AI ecosystem to build regulatory and audit know-how.
• Establish the intended use for your medical device
• Determine the device classification for your healthcare software
• Select an approved Notified Body to perform the conformity assessment
• Prepare the documentation for the QMS audit and prepare employees for onsite QMS checks. Undergo onsite visits and QMS audits. Receive ISO 13485 certification
• Prepare the documentation for the Technical File, including the Clinical Evaluation Report, and submit it
• Sign and submit the Declaration of Conformity
• Undergo Technical Files audit by Notified Body
• Following approval, adhere to the requirements for affixing the CE mark to your device.
  

‍Special thanks to Leander Maerkisch, Founder and Managing Director of Floy, for providing insight and expertise to this article.

Encord is a medical imaging annotation tool and active learning platform for computer vision. It is fully auditable and features SDK and API access, making it the perfect annotation tool for building CE-compliant diagnostic models. Get a demo of the platform to understand more about how Encord can help you.