Qualitative Risk Analysis

Risk can be associated with a number of different types of consequences, impacting different objectives. The types of consequences to be analyzed are decided when planning the assessment. The context statement is checked to ensure that the consequences to be analyzed align with the purpose of the assessment and the decisions to be made. This can be revisited during the assessment as more is learned.

Methods used in analyzing risks can be qualitative, semiquantitative, or quantitative. The decision here will be on the intended use, the availability of reliable data, and the decision-making needs of the organization. In ICH Q9 this is also the level of formality.

Risk Is….
The combination of the probability of the occurrence of the harm and the severity of that harm.	The effect of uncertainty on objectives	Often characterized by reference to the potential event and consequences or combination of these	Often expressed in terms of a combination of the consequences of an event (including in changes in circumstances) and the associated likelihood of the occurrence

Qualitative assessments define consequence (or severity), likelihood, and level of risk by significance levels, such as “high,” “medium,” or “low.” They work best when supporting analysis that have a narrow application or are within another quality system, such as change control.

Below is a good way to break down consequences and likelihood for a less formal assessment.

	Consequence				Increase Likelihood
Severity	People	Assets	Requirements	Ability to Meet Regulations	Never Heard of in Industry	B. Has Occurred in Industry	C. Occurs Several Times Per Year in Company	D. Occurs Several Times Per Year at Location
0	No Injury	No Damage	No Effect	No Impact	Manage for Continuous Improvement
1	Slight Injury	Slight Damage	Slight Effect	Slight Impact			Incorporate Risk – Reduction Measures
2	Minor Injury	Minor Damage	Limited Effect	Limited Impact			Incorporate Risk – Reduction Measures
3	Major Injury	Localized Damage	Localized Effect	Considerable Impact			Intolerable – Immediate Corrective Action
4	1-3 Fatalities	Major Damage	Major Effect	National Impact
5	Multiple Fatalities	Extensive Damage	Massive Effect	International Impact

Treating All Investigations the Same

Stephanie Gaulding, a colleague in the ASQ, recently wrote an excellent post for Redica on “How to Avoid Three Common Deviation Investigation Pitfalls“, a subject near and dear to my heart.

The three pitfalls Stephanie gives are:

Not getting to root case
Inadequate scoping
Treating investigations the same

All three are right on the nose, and I’ve posted a bunch on the topics. Definitely go and read the post.

What I want to delve deeper into is Stephanie’s point that “Deviation systems should also be built to triage events into risk-based categories with sufficient time allocated to each category to drive risk-based investigations and focus the most time and effort on the highest risk and most complex events.”

That is an accurate breakdown, and exactly what regulators are asking for. However, I think the implementation of risk-based categories can sometimes lead to confusion, and we can spend some time unpacking the concept.

Risk is the possible effect of uncertainty. Risk is often described in terms of risk sources, potential events, their consequences, and their likelihoods (where we get likelihoodXseverity from).

But there are a lot of types of uncertainty, IEC31010 “Risk management – risk management techniques” lists the following examples:

uncertainty as to the truth of assumptions, including presumptions about how people or systems might behave
variability in the parameters on which a decision is to be based
uncertainty in the validity or accuracy of models which have been established to make predictions about the future
events (including changes in circumstances or conditions) whose occurrence, character or consequences are uncertain
uncertainty associated with disruptive events
the uncertain outcomes of systemic issues, such as shortages of competent staff, that can have wide ranging impacts which cannot be clearly defined lack of knowledge which arises when uncertainty is recognized but not fully understood
unpredictability
uncertainty arising from the limitations of the human mind, for example in understanding complex data, predicting situations with long-term consequences or making bias-free judgments.

Most of these are only, at best, obliquely relevant to risk categorizing deviations.

So it is important to first build the risk categories on consequences. At the end of the day these are the consequence that matter in the pharmaceutical/medical device world:

harm to the safety, rights, or well-being of patients, subjects or participants (human or non-human)
compromised data integrity so that confidence in the results, outcome, or decision dependent on the data is impacted

These are some pretty hefty areas and really hard for the average user to get their minds around. This is why building good requirements, and understanding how systems work is so critical. Building breadcrumbs in our procedures to let folks know what deviations are in what category is a good best practice.

There is nothing wrong with recognizing that different areas have different decision trees. Harm to safety in GMP can mean different things than safety in a GLP study.

The second place I’ve seen this go wrong has to do with likelihood, and folks getting symptom confused with problem confused with cause.

All deviations are with a situation that is different in some way from expected results. Deviations start with the symptom, and through analysis end up with a root cause. So when building your decision-tree, ensure it looks at symptoms and how the symptom is observed. That is surprisingly hard to do, which is why a lot of deviation criticality scales tend to focus only on severity.

Success/Failure Space, or Why We Can Sometimes Seem Pessimistic

When evaluating a system we can look at it in two ways. We can identify ways a thing can fail or the various ways it can succeed.

These are really just two sides of the coin in many ways, with identifiable points in success space coinciding with analogous points in failure space. “Maximum anticipated success” in success space coincides with “minimum anticipated failure” in failure space.

Like everything, how we frame the question helps us find answers. Certain questions require us to think in terms of failure space, others in success. There are advantages in both, but in risk management, the failure space is incredibly valuable.

It is generally easier to attain concurrence on what constitutes failure than it is to agree on what constitutes success. We may desire a house that has great windows, high ceilings, a nice yard. However, the one we buy can have a termite-infested foundation, bad electrical work, and a roof full of leaks. Whether the house is great is a matter of opinion, but we certainly know all it is a failure based on the high repair bills we are going to accrue.

Success tends to be associated with the efficiency of a system, the amount of output, the degree of usefulness. These characteristics are describable by continuous variables which are not easily modeled in terms of simple discrete events, such as “water is not hot” which characterizes the failure space. Failure, in particular, complete failure, is generally easy to define, whereas the event, success, maybe more difficult to tie down

Theoretically the number of ways in which a system can fail and the number of ways in which a system can ·succeed are both infinite, from a practical standpoint there are generally more ways to success than there are to failure. From a practical point of view, the size of the population in the failure space is less than the size of the population in the success space. This leads to risk management focusing on the failure space.

The failure space maps really well to nominal scales for severity, which can be helpful as you build your own scales for risk assessments.

For example, let’s look at an example of a morning commute.