Note: this article is a summary of the original article published on my Let's Talk Risk! newsletter. Read the full article here.

Summary

This case study provides an easy way to estimate the probability of occurrence of harm (POH) using post-market data, and to evaluate changes in the risk level.

Problem Statement

A common challenge during the post-market phase of a medical device is to determine if there is a change in previously estimated risk levels, and if any individual risk is no longer acceptable based on pre-determined criteria for risk acceptability. This is a key requirement of ISO 14971:2019 in Clause 10.3.

Industry practice is to monitor changes in the frequency of complaints and adverse events to look for unusual trends or outliers. Some of the commonly used techniques include trend analysis, bar charts, pareto analysis and qualitative methods.

Risk, however is a defined as the combination of probability of occurrence of harm (POH) and the severity of that harm. There is no direct way to estimate changes in the risk level by only monitoring the frequency of events. Another challenge is that there is no direct link between risk levels estimated in an FMEA (Failure Modes and Effects Analysis) and complaints and adverse events reported during the post-market phase.

What if we could develop a method to directly estimate a probability of occurrence of harm (POH) associated with device malfunctions using actual post-market data?

This could not only help us monitor changes in the risk level for an existing device, but it could also provide a set of baseline data to use in the development of the next generation of devices.

Case Study

This case study is based on publicly available information about the HeartWare Ventricular Assist Device (HVAD).

The HVAD system is indicated for patients with end-stage heart failure, either as a bridge therapy to heart transplant (BTT), or as destination therapy (DT). As shown in the following figure, it includes an implantable mechanical pump to drive blood from the left ventricle to the body, a driveline that connects to an external controller, dual power sources and a data monitor.

Let us review a few basic concepts and common challenges

Figure C.1 in Annex C in ISO 14971:2019 provides one method of estimating the probability of occurrence of harm (POH) as the product of P1 and P2.

As shown above, P1 is the probability of occurrence of a hazardous situation, and P2 is the probability of a hazardous situation leading to harm.

It is important to note that P2 is a conditional probability, applicable only when a hazardous situation has occurred.

In practice, it is very challenging to accurately estimate P1 and P2 values.

A common industry practice is to use an FMEA as the only method of risk analysis in the context of ISO 14971. However, a single failure mode/cause and effect item in an FMEA is not necessarily a direct representation of a hazardous situation. Therefore, the probability of a specific failure mode/cause is not the same as P1. This is a common error in risk analysis where a specific failure modes are assumed to directly represent hazardous situations. Clearly, there is more to hazard analysis than FMEA alone!

P2, on the other hand, is even more challenging to estimate because there is very little data available linking a hazardous situation with specific harms. A common industry practice is to estimate P2 based on prior clinical experience.

A simple method of estimating P1, P2 and POH and evaluating changes in the risk level

The following Figure summarizes the information used in this case study to illustrate a simple method of estimating P1, P2 and POH from post-market data.

If we make an assumption that delayed restart, or failure to restart, presents a hazardous situation, we can use the pump failure rate to estimate P1. Note that delayed restart, or failure to restart, may be an effect of multiple failure modes at different levels of the system hierarchy. However, since an overall failure rate is already available, we can use to estimate P1.

For example, a failure rate of 5.7% for pumps with defective part translates to P1 = 0.057. When operating in the normal mode, with dual stator, the failure rate of 0.087% translates to P1 = 0.00087.

As shown in the table in Figure 3, there are 26 events related to harms of different severity. P2 for each harm severity can be estimated as the number of events for that harm divided by the total number of events.

For example, 2 out of 26 events resulted in death. P2 for death, therefore, can be estimated as 2/26 = 0.077.

Now, using the formula POH = P1*P2, we can estimate the probability of death due to delayed restart or failure to restart when a pump with defective part is used as POH = 0.057*0.077 = 0.0044, or about 4 in 1000.

In the same manner, we can estimate probability of death in normal mode for general pump population as POH = 0.00087*0.057 = 0.000067, or about 7 in 100,000.

As illustrated above, we can see that the risk of death has increased by 2 orders of magnitude for pumps assembled with defective parts.

A similar calculation can be done for harms of other severities as discussed in the full article here.

Limitations

Although this method of estimating P1, P2 and POH is simple, it is not without limitations. Here are some of these limitations we should be aware of:

• Generally, there is a lot of variability in complaints reporting. As a result, it is challenging to establish baseline failure rates for estimating P1 and P2.
• Estimating P1 and P2 values using complaints data is highly dependent on the time interval used for data analysis. As an example, the above data set corresponds to a time interval of more than 3 years. Estimated values for P1 and P2 may vary widely across shorter time intervals.
• Often, there is no direct link between a device failure and reported patient harm. As a result, it is challenging to estimate baseline P2 values. A common practice is to utilize subject matter expertise from clinical/medical experts to create a table of baseline P2 values corresponding to each of the harm severity levels. However, these baseline values need to be updated on a periodic basis using complaints data.
• Often, detailed information about the sequence of events leading to the occurrence of a hazardous situation is not available from complaints data. As a result, P1 values may be overestimated if the failure rate is used as a direct measure of P1.
• P2 values may be under or over-estimated, or unevenly distributed among harms of different severity levels due to variability in complaints reporting, and severity level assignments. This issue become more serious if standardized harm terms are not used to classify incoming complaints.

Key Points

• If a direct link between device failure and patient harm is available from complaints data, we can estimate P1 and P2 values for different types of harms.
• We can estimate P1 using device failure rates, assuming the combined probability of the sequence of events as 1.
• We can estimate P2 values for each harm severity level, if we have a full distribution of harm events across all severity levels. P2 is the conditional probability applicable to a hazardous situation. As a practical matter, P2 values can be distributed across different harm severity levels, but they must all add up to 1 for a given hazardous situation.
• The probability of occurrence (POH) of harm at each severity level can be calculated as the product of P1 and P2 corresponding to that severity level.
• Changes in the POH level can be mapped on a risk matrix to compare them against a baseline to determine the risk of harm has changed to an unacceptable level.
• Risk of harm should be evaluated on a periodic basis using post-market surveillance information. As much as possible, a quantitative approach should be used to estimate changes in the risk level.

References

1. FDA: HVAD Summary of Safety and Effectiveness Data (SSED)
2. Medtronic Urgent Medical Device Safety Communication, December 2020
3. Full article: Using post-market data to evaluate changes in risk level