The Food and Drug Administration (FDA or the Agency), the US regulating authority in the sphere of medical devices and other healthcare products, has published a guidance document dedicated to software validation. The document provides an overview of the main activities to be carried out in the course of software validation including, inter alia, testing by the software developer and user site testing. It is important to mention that the present guidance is not intended to introduce new rules and requirements but to provide additional clarifications and recommendations to be considered by medical device manufacturers (software developers) and other parties involved. Moreover, an alternative approach could be applied, provided such an approach complies with the respective regulatory requirements and has been agreed with the Agency in advance.
Table of Contents
Testing by the Software Developer: Key Points
According to the guidance, the software testing process stands for running software products under known conditions with defined inputs and documented outcomes that can be compared to their predefined expectations. The FDA further mentions that due to the complexity of the process and significant time needed to complete it, software testing should be subject to rigorous planning to ensure its effectiveness. Thus, the authority encourages medical device manufacturers (software developers) to create software testing plans as early as possible. Such plans should cover the main aspects related to the future software testing including, inter alia, such factors as schedules, environments, resources, methodologies, cases, documentation, and reporting criteria. The efforts required would depend on the complexity of the software product subject to testing, as well as its intended purpose and risks associated thereto in terms of potential failures. The guidance also provides additional references to be considered, namely:
- NIST Special Publication 500-235, Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric;
- NUREG/CR-6293, Verification and Validation Guidelines for High Integrity Systems; and
- IEEE Computer Society Press, Handbook of Software Reliability Engineering.
As explained by the FDA, the plans to be duly developed by the manufacturer should outline the specific tasks associated with each development stage, as well as respective completion criteria.
Class I Software Medical Devices
FDA’s Class I is the lowest class under the applicable classification system and applies to the devices with the lowest risk associated thereto. Hence, the applicable regulatory controls are quite low as well. In the case of software-based medical devices, this class applies for the devices that are:
- Intended to monitor the state or progression of a disease;
- Providing information that does not indicate if an individual may be in danger;
- Associated with a low public health risk.
Class IIa Software Medical Devices
This category applies to medical devices associated with medium risk. An example provided by the TGA describes diabetes diagnosis software that is intended to be used by a healthcare professional. Thus, such a product is a Class IIa medical device as the device provides information to a relevant health professional to inform the diagnosis of a serious disease. This category also covers risk prediction software, as well as the tools that record data from a patient monitor or images directly from an MRI scanner (provided that such software does not impact the operations of a scanner itself).
Class IIb Software Medical Devices
This category applies to medical devices associated with medium-high risk. For instance, such classification should be applied to a product that is intended to analyze a cardiac MRI in order to provide information used in making diagnoses of related diseases. As in the previous example, the software is intended to provide information to healthcare professionals only. As described in the guidance, Class IIb applies to medical software that is intended by the manufacturer (the software developer) to provide information to a relevant health professional to inform the diagnosis of a serious disease. Other examples of Class IIb products include tools intended to be used to diagnose an acute arterial occlusion due to the severity of potential consequences of this disease if the necessary treatment is not applied. This category also covers software products that are intended to provide recommendations for treatment or intervention on the basis of input data (e.g., a coronary angiogram). As in the previous cases, such software should be used only by healthcare professionals. Consequently, a Class IIb software-based medical device is the one that is intended to:
- Recommend a treatment or intervention to a relevant health professional for the purposes of making a decision about the treatment or intervention; and
- Be used in cases when the absence of a treatment or a treatment itself could result in severe health deterioration or other adverse consequences.
The same classification applies to wearable devices intended to collect and analyze data for screening for serious heart diseases, as well as questionnaire apps intended to analyze the information provided by a patient and provide a diagnostic output.
Testing Coverage
The FDA also acknowledges that most of the software products, apart from the simplest ones, could not be exhaustively tested, as it is impossible to include all potential inputs and get all potential outputs. Hence, none of the testing methodologies covers any of the issues that may arise. If in the course of testing no errors were identified, this should not be construed as that there are no errors in the software at all. Thus, medical device manufacturers (software developers) should be aware that irrespectively of how rigorous the testing was, a chance of error still exists.
One of the main concepts associated with software testing is an expected result, which should be later compared to an actual result to evaluate how the software performs. All important aspects should be covered by the respective specification. According to the guidance, a software specification document must identify what, when, how, why, etc., is to be achieved with an engineering (i.e., measurable or objectively verifiable) level of detail for it to be confirmed through testing. Consequently, the actual scope of testing and the approach to be applied should depend on the particular aspects subject to testing.
To ensure the effectiveness of the testing process, it should be based on the appropriate principles. The FDA guidance outlines the main software principles to be considered, namely:
- The expected test outcome is predefined;
- A good test case has a high probability of exposing an error;
- A successful test finds an error;
- Coding is independent;
- Both application (user) and software (programming) expertise are employed;
- Testers use different tools from coders;
- Examining only the usual case is insufficient;
- Testing documentation permits its reuse and independent confirmation of the pass/fail status of a test outcome during the subsequent review.
Under the general rule, the software testing process commences upon completion of code inspection. In terms of testing levels, the manufacturer should start from specific units and then move to the system in general. The scope of testing conducted should cover all important aspects to evaluate compliance with the requirements outlined in the applicable specifications.
Software Testing: Types and Approaches
Some of the testing methods are based on the knowledge of the source code and information deriving thereof – so-called “white-box” testing. Such tests are intended to evaluate the way the program makes a decision based on the predefined algorithm. In most cases, such testing is used at a unit (module) level. However, it could be also used for other levels.
The kind of testing described hereinabove also refers to “structural testing”. In the case of such testing, its level could be described as a percentage of the covered share to the whole structure. As described by the FDA, metrics that are usually used concerning structural coverage include, inter alia, the following ones:
- Statement Coverage;
- Decision (Branch) Coverage;
- Condition Coverage;
- Multi-Condition Coverage;
- Loop Coverage;
- Path Coverage;
- Data Flow Coverage.
Another type of testing described in the present guidance is definition-based or specification-based testing, which is also referred to as “black-box” testing. According to the guidance, the testing of this type identifies test cases based on the definition of what the software product (whether it be a unit (module) or a complete program) is intended to do; these test cases challenge the intended use or functionality of a program and the program’s internal and external interfaces. As in the case with structural testing, functional testing could be also applied to all levels from separate units (modules) to the whole system.
According to the guidance, there are several types of functional testing, namely:
- Normal Case;
- Output Forcing;
- Robustness;
- Combination of Inputs.
Both structural and functional testing methods are based on the use of specific inputs. However, neither of them ensures the product’s reliability. To expand the evaluation scope, statistical testing could be applied. This approach is based on the use of input data which is randomly generated for testing purposes by the intended use. Moreover, the input data could be generated to address specific concerns that are of importance for the particular software. As described by the FDA, this type of testing allows identifying operating conditions that occur quite rarely and were not initially anticipated. A combination of all three methods described hereinabove allows to expand the testing scope to the maximum extent possible to ensure the most important aspects are covered, and most of the potential issues would be duly identified.
In summary, the present FDA guidance covers the key points related to software testing to be carried out by the software developer as a part of a software validation process. The document describes in detail existing types of testing and outlines their strengths and weaknesses.
Sources:
How Can RegDesk Help?
RegDesk is a next-generation web-based software for medical device and IVD companies. Our cutting-edge platform uses machine learning to provide regulatory intelligence, application preparation, submission, and approvals management globally. Our clients also have access to our network of over 4000 compliance experts worldwide to obtain verification on critical questions. Applications that normally take 6 months to prepare can now be prepared within 6 days using RegDesk Dash(TM). Global expansion has never been this simple.
