Tag: uncertainty

Assessing the Strength of Knowledge: A Framework for Decision-Making

ICH Q9(R1) emphasizes that knowledge is fundamental to effective risk management. The guideline states that “QRM is part of building knowledge and understanding risk scenarios, so that appropriate risk control can be decided upon for use during the commercial manufacturing phase.” 

We need to recognize the inverse relationship between knowledge and uncertainty in risk assessment. ICH Q9(R1) notes that uncertainty may be reduced “via effective knowledge management, which enables accumulated and new information (both internal and external) to be used to support risk-based decisions throughout the product lifecycle”

In order to gauge the confidence in risk assessment we need to gauge our knowledge strength.

The Spectrum of Knowledge Strength

Knowledge strength can be categorized into three levels: weak, medium, and strong. Each level is determined by specific criteria that assess the reliability, consensus, and depth of understanding surrounding a particular subject.

Indicators of Weak Knowledge

Knowledge is considered weak if it exhibits one or more of the following characteristics:

  1. Oversimplified Assumptions: The foundations of the knowledge rely on strong simplifications that may not accurately represent reality.
  2. Lack of Reliable Data: There is little to no data available, or the existing information is highly unreliable or irrelevant.
  3. Expert Disagreement: There is significant disagreement among experts in the field.
  4. Poor Understanding of Phenomena: The underlying phenomena are poorly understood, and available models are either non-existent or known to provide inaccurate predictions.
  5. Unexamined Knowledge: The knowledge has not been thoroughly scrutinized, potentially overlooking critical “unknown knowns.”

Hallmarks of Strong Knowledge

On the other hand, knowledge is deemed strong when it meets all of the following criteria (where relevant):

  1. Reasonable Assumptions: The assumptions made are considered very reasonable and well-grounded.
  2. Abundant Reliable Data: Large amounts of reliable and relevant data or information are available.
  3. Expert Consensus: There is broad agreement among experts in the field.
  4. Well-Understood Phenomena: The phenomena involved are well understood, and the models used provide predictions with the required accuracy.
  5. Thoroughly Examined: The knowledge has been rigorously examined and tested.

The Middle Ground: Medium Strength Knowledge

Cases that fall between weak and strong are classified as medium strength knowledge. This category can be flexible, allowing for a broader range of scenarios to be considered strong. For example, knowledge could be classified as strong if at least one (or more) of the strong criteria are met while none of the weak criteria are present.

Strong vs Weak Knowledge

A Simplified Approach

For practical applications, a simplified version of this framework can be used:

  • Strong: All criteria for strong knowledge are met.
  • Medium: One or two criteria for strong knowledge are not met.
  • Weak: Three or more criteria for strong knowledge are not met.

Implications for Decision-Making

Understanding the strength of our knowledge is crucial for effective decision-making. Strong knowledge provides a solid foundation for confident choices, while weak knowledge signals the need for caution and further investigation.

When faced with weak knowledge:

  • Seek additional information or expert opinions
  • Consider multiple scenarios and potential outcomes
  • Implement risk mitigation strategies

When working with strong knowledge:

  • Make decisions with greater confidence
  • Focus on implementation and optimization
  • Monitor outcomes to validate and refine understanding

Knowledge Strength and Uncertainty

The concept of knowledge strength aligns closely with the levels of uncertainty.

Strong Knowledge and Low Uncertainty (Levels 1-2)

Strong knowledge typically corresponds to lower levels of uncertainty:

  • Level 1 Uncertainty: This aligns closely with strong knowledge, where outcomes can be estimated with reasonable accuracy within a single system model. Strong knowledge is characterized by reasonable assumptions, abundant reliable data, and well-understood phenomena, which enable accurate predictions.
  • Level 2 Uncertainty: While displaying alternative futures, this level still operates within a single system where probability estimates can be applied confidently. Strong knowledge often allows for this level of certainty, as it involves broad expert agreement and thoroughly examined information.

Medium Knowledge and Moderate Uncertainty (Level 3)

Medium strength knowledge often corresponds to Level 3 uncertainty:

  • Level 3 Uncertainty: This level involves “a multiplicity of plausible futures” with multiple interacting systems, but still within a known range of outcomes. Medium knowledge strength might involve some gaps or disagreements but still provides a foundation for identifying potential outcomes.

Weak Knowledge and Deep Uncertainty (Level 4)

Weak knowledge aligns most closely with the deepest level of uncertainty:

  • Level 4 Uncertainty: This level leads to an “unknown future” where we don’t understand the system and are aware of crucial unknowns. Weak knowledge, characterized by oversimplified assumptions, lack of reliable data, and poor understanding of phenomena, often results in this level of deep uncertainty.

Implications for Decision-Making

  1. When knowledge is strong and uncertainty is low (Levels 1-2), decision-makers can rely more confidently on predictions and probability estimates.
  2. As knowledge strength decreases and uncertainty increases (Levels 3-4), decision-makers must adopt more flexible and adaptive approaches to account for a wider range of possible futures.
  3. The principle that “uncertainty should always be considered at the deepest proposed level” unless proven otherwise aligns with the cautious approach of assessing knowledge strength. This ensures that potential weaknesses in knowledge are not overlooked.

Conclusion

By systematically evaluating the strength of our knowledge using this framework, we can make more informed decisions, identify areas that require further investigation, and better understand the limitations of our current understanding. Remember, the goal is not always to achieve perfect knowledge but to recognize the level of certainty we have and act accordingly.

Risk Management Addresses Uncertainty

The ICH Q9 guideline on Quality Risk Management (QRM), including its revised version ICH Q9(R1), addresses the concept of uncertainty as a critical component in risk management within the pharmaceutical industry.

Understanding Uncertainty in ICH Q9

Uncertainty in the context of ICH Q9 refers to the lack of complete knowledge about a process and its expected or unexpected variability. This uncertainty can stem from various sources, including gaps in knowledge about pharmaceutical science, process understanding, and potential failure modes.

Key Points on Uncertainty from ICH Q9(R1)

Sources of Uncertainty:

    • Knowledge Gaps: Incomplete understanding of the scientific and technical aspects of processes.
    • Process Variability: Both expected and unexpected changes in process performance.
    • Failure Modes: Unidentified or poorly understood potential points of failure in processes or systems.

    Managing Uncertainty:

      • Risk-Based Decision Making: The guideline emphasizes that decisions should be made based on the level of uncertainty, importance, and complexity of the situation. This means that more formal and structured approaches should be used when uncertainty is high.
      • Formality in QRM: ICH Q9(R1) introduces the concept of formality as a spectrum, suggesting that the degree of formality in risk management activities should be commensurate with the level of uncertainty. Less formal methods may be appropriate for well-understood processes, while highly structured methods are necessary for areas with high uncertainty.

      Reducing Subjectivity:

        • The guideline acknowledges that subjectivity can impact the effectiveness of risk management. It recommends strategies to minimize subjectivity, such as using well-recognized risk assessment tools and involving cross-functional teams to provide diverse perspectives.

        Continuous Improvement:

          • ICH Q9(R1) stresses the importance of continual improvement in risk management processes. This involves regularly updating risk assessments and control measures as new information becomes available, thereby reducing uncertainty over time.

          Practical Implementation

          In practice, managing uncertainty within the framework of ICH Q9 involves:

          • Conducting thorough risk assessments to identify potential hazards and their associated risks.
          • Applying appropriate risk control measures based on the level of uncertainty and the criticality of the process.
          • Documenting and reviewing risk management activities to ensure they remain relevant and effective as new information is obtained.

          Conclusion

          The ICH Q9 approach to uncertainty underscores the importance of a structured, knowledge-based approach to risk management in the pharmaceutical industry. By addressing uncertainty through rigorous risk assessments and appropriate control measures, organizations can enhance the reliability and safety of their processes and products, ultimately safeguarding patient health and safety.

          Q9 (r1) Risk Management Draft

          Q9 (r1) starts with all the same sections on scope and purpose. There are slight differences in ordering in scope, mainly because of the new sections below, but there isn’t much substantially different.

          4.1 Responsibilities

          This is the first major change with added paragraphs on subjectivity, which basically admits that it exists and everyone should be aware of that. This is the first major change that should be addressed in the quality system “All participants involved with quality risk management activities should acknowledge, anticipate, and address the potential for subjectivity.”

          Aligned with that requirement is a third bullet for decision-makers: “assure that subjectivity in quality risk management activities is controlled and minimised, to facilitate scientifically robust risk-based decision making.”

          Solid additions, if a bit high level. A topic of some interest on this blog, recognizing the impact of subjectivity is critical to truly developing good risk management.

          Expect to start getting questions on how you acknowledge, anticipate and address subjectivity. It will take a few years for this to work its way through the various inspectorates after approval, but it will. There are various ways to crack this, but it will require both training and tools to make it happen. It also reinforces the need for well-trained facilitators.

          5.1 Formality in Quality Risk Management

          “The degree of rigor and formality of quality risk management should reflect available knowledge and be commensurate with the complexity and/ or criticality of the issue to be addressed.”

          That statement in Q9 has long been a nugget of long debate, so it is good to see section 5.1 added to give guidance on how to implement it, utilizing 3 axis:

          • Uncertainty: This draft of Q9 utilizes a fairly simple definition of uncertainty and needs to be better aligned to ISO 31000. This is where I am going to definitely submit comments. Taking a straight knowledge management approach and defining uncertainty solely on lack of knowledge misses the other element of uncertainty that are important.
          • Importance: This was probably the critical determination folks applied to formality in the past.
          • Complexity: Not much said on complexity, which is worrisome because this is a tough one to truly analyze. It requires system thinking, and a ot of folks really get complicated and complex confused.

          This section is important, the industry needs it as too many companies have primitive risk management approaches because they shoe-horn everything into a one size fits all level of formality and thus either go overboard or do not go far enough. But as written this draft of Q9 is a boon to consultants.

          We then go on to get just how much effort should go into higher formality versus lower level of formality which boils down to higher formality is more stand alone and lower formality happens within another aspect of the quality system.

          5.2 Risk-based Decision Making

          Another new section, definitely designed to align to ISO 9001-2015 thinking. Based on the level of formality we are given three types with the first two covering separate risk management activities and the third being rule-based in procedures.

          6. INTEGRATION OF QUALITY RISK MANAGEMENT INTO INDUSTRY AND REGULATORY OPERATIONS

          Section 6 gets new subsection “The role of Quality Risk Management in addressing Product Availability Risks,” “Manufacturing Process Variation and State of Control (internal and external),” “Manufacturing Facilities,” “Oversight of Outsourced Activities and Suppliers.” These new subsections expand on what used to be solely a list of bullet points and provide some points to consider in their topic area. They are also good things to make sure risk management is built into if not already there.

          Overall Thoughts

          The ICH members did exactly what they told us they were going to do, and pretty much nothing else. I do not think they dealt with the issues deeply and definitively enough, and have added a whole lot of ambiguity into the guidance. which is better than being silent on the topic, but I’m hoping for a lot more.

          Subjectivity, uncertainty, and formality are critical topics. Hopefully your risk management program is already taking these into account.

          I’m hoping we will also see a quick revision of the PIC/S “Assessment of Quality Risk Management Implementation” to align to these concepts.

          Information Gaps

          An information gap is a known unknown, a question that one is aware of but for which one is uncertain of the answer. It is a disparity between what the decision maker knows and what could be known The attention paid to such an information gap depends on two key factors: salience, and importance.

          • The salience of a question indicates the degree to which contextual factors in a situation highlight it. Salience might depend, for example, on whether there is an obvious counterfactual in which the question can be definitively answered.
          • The importance of a question is a measure of how much one’s utility would depend on the actual answer. It is this factor—importance—which is influenced by actions like gambling on the answer or taking on risk that the information gap would be relevant for assessing.

          Information gaps often dwell in the land of knightian uncertainty.

          Communicating these Known Unknowns

          Communicating around Known Unknowns and other forms of uncertainty

          A wide range of reasons for information gaps exist:

          • variability within a sampled population or repeated measures leading to, for example, statistical margins-of-error
          • computational or systematic inadequacies of measurement
          • limited knowledge and ignorance about underlying processes
          • expert disagreement.