“Advice for the government on communicating uncertainty” by Jessica Hullman on the blog Statistical Modeling, Causal Inference, and Social Science is a great read.
Q9 (r1) starts with all the same sections on scope and purpose. There are slight differences in ordering in scope, mainly because of the new sections below, but there isn’t much substantially different.
This is the first major change with added paragraphs on subjectivity, which basically admits that it exists and everyone should be aware of that. This is the first major change that should be addressed in the quality system “All participants involved with quality risk management activities should acknowledge, anticipate, and address the potential for subjectivity.”
Aligned with that requirement is a third bullet for decision-makers: “assure that subjectivity in quality risk management activities is controlled and minimised, to facilitate scientifically robust risk-based decision making.”
Expect to start getting questions on how you acknowledge, anticipate and address subjectivity. It will take a few years for this to work its way through the various inspectorates after approval, but it will. There are various ways to crack this, but it will require both training and tools to make it happen. It also reinforces the need for well-trained facilitators.
5.1 Formality in Quality Risk Management
“The degree of rigor and formality of quality risk management should reflect available knowledge and be commensurate with the complexity and/ or criticality of the issue to be addressed.”
That statement in Q9 has long been a nugget of long debate, so it is good to see section 5.1 added to give guidance on how to implement it, utilizing 3 axis:
- Uncertainty: This draft of Q9 utilizes a fairly simple definition of uncertainty and needs to be better aligned to ISO 31000. This is where I am going to definitely submit comments. Taking a straight knowledge management approach and defining uncertainty solely on lack of knowledge misses the other element of uncertainty that are important.
- Importance: This was probably the critical determination folks applied to formality in the past.
- Complexity: Not much said on complexity, which is worrisome because this is a tough one to truly analyze. It requires system thinking, and a ot of folks really get complicated and complex confused.
This section is important, the industry needs it as too many companies have primitive risk management approaches because they shoe-horn everything into a one size fits all level of formality and thus either go overboard or do not go far enough. But as written this draft of Q9 is a boon to consultants.
We then go on to get just how much effort should go into higher formality versus lower level of formality which boils down to higher formality is more stand alone and lower formality happens within another aspect of the quality system.
5.2 Risk-based Decision Making
Another new section, definitely designed to align to ISO 9001-2015 thinking. Based on the level of formality we are given three types with the first two covering separate risk management activities and the third being rule-based in procedures.
6. INTEGRATION OF QUALITY RISK MANAGEMENT INTO INDUSTRY AND REGULATORY OPERATIONS
Section 6 gets new subsection “The role of Quality Risk Management in addressing Product Availability Risks,” “Manufacturing Process Variation and State of Control (internal and external),” “Manufacturing Facilities,” “Oversight of Outsourced Activities and Suppliers.” These new subsections expand on what used to be solely a list of bullet points and provide some points to consider in their topic area. They are also good things to make sure risk management is built into if not already there.
The ICH members did exactly what they told us they were going to do, and pretty much nothing else. I do not think they dealt with the issues deeply and definitively enough, and have added a whole lot of ambiguity into the guidance. which is better than being silent on the topic, but I’m hoping for a lot more.
Subjectivity, uncertainty, and formality are critical topics. Hopefully your risk management program is already taking these into account.
I’m hoping we will also see a quick revision of the PIC/S “Assessment of Quality Risk Management Implementation” to align to these concepts.
An information gap is a known unknown, a question that one is aware of but for which one is uncertain of the answer. It is a disparity between what the decision maker knows and what could be known The attention paid to such an information gap depends on two key factors: salience, and importance.
- The salience of a question indicates the degree to which contextual factors in a situation highlight it. Salience might depend, for example, on whether there is an obvious counterfactual in which the question can be definitively answered.
- The importance of a question is a measure of how much one’s utility would depend on the actual answer. It is this factor—importance—which is influenced by actions like gambling on the answer or taking on risk that the information gap would be relevant for assessing.
Information gaps often dwell in the land of knightian uncertainty.
Communicating these Known Unknowns
A wide range of reasons for information gaps exist:
- Level 1 uncertainties are defined as relatively minor – as representing “a clear enough future” set within a “single system model” whereby outcomes can be estimated with reasonable accuracy;
- Level 2 uncertainties display “alternative futures” but, again, within a single system in which probability estimates can be applied with confidence.
Levels 3 and 4 uncertainties are described as representing “deep uncertainty”.
- Level 3 uncertainties are described as “a multiplicity of plausible futures”, in which multiple systems interact, but in which we can identify “a known range of outcomes”
- Level 4 uncertainties lead us to an “unknown future” in which we don’t understand the system: we know only that there is something, or are some things, that we know we don’t know.
This hierarchy can be useful to help us think carefully about whether the uncertainty behind a problem can be defined in terms of a Level 1 prediction, with parameters for variation. Or, can it be resolved as group of Level 2 possibilities with probability estimates for each? Can the issue only be understood as a set of different Level 3 futures, each with a clear set of defined outcomes, or only by means of a Level 4 statement to the effect that we know only that there is something crucial that we don’t yet know?
There is often no clear or unanimous view of whether a particular uncertainty is set at a specific level. Uncertainty should always be considered at the deepest proposed level, unless or until those that propose this level can be convinced by an evidence-based argument that it should be otherwise.
- Walker, W.E., Marchau, V.A.W.J. and Swanson, D. (2010) “Addressing Deep Uncertainty using Adaptive Policies: Introduction to Section 2”, Technological Forecasting & Social Change, 77: 917–23.
An important part of innovation, risk management, change management, continuous improvement is overcoming the fear of the unknown. We humans are wired with an intense aversion to both risk and uncertainty. Research shows that both have separate neural reactions and that choices with ambiguous outcomes trigger a stronger fear response than do risky choices. Additional research shows that the risk itself isn’t so much the problem, but the uncertainty is: we are afraid primarily because we don’t know the outcome and less so because of the risk.
There are three types of uncertainty:
- Aleatoric Uncertainty: The uncertainty of quantifiable probabilities.
- Epistemic Uncertainty: The uncertainty of knowledge.
- Knightian Uncertainty: The uncertainty of nonquantifiable risk.
I wrote more on this in my post “Uncertainty and Subjectivity in Risk Management.” This post mostly stems from wanting an excuse to share a funny comic.