Beyond “Knowing Is Half the Battle”

Dr. Valerie Mulholland’s recent exploration of the GI Joe Bias strikes gets to the heart of a fundamental challenge in pharmaceutical quality management: the persistent belief that awareness of cognitive biases is sufficient to overcome them. I find Valerie’s analysis particularly compelling because it connects directly to the practical realities we face when implementing ICH Q9(R1)’s mandate to actively manage subjectivity in risk assessment.

Valerie’s observation that “awareness of a bias does little to prevent it from influencing our decisions” shows us that the GI Joe Bias underlays a critical gap between intellectual understanding and practical application—a gap that pharmaceutical organizations must bridge if they hope to achieve the risk-based decision-making excellence that ICH Q9(R1) demands.

The Expertise Paradox: Why Quality Professionals Are Particularly Vulnerable

Valerie correctly identifies that quality risk management facilitators are often better at spotting biases in others than in themselves. This observation connects to a deeper challenge I’ve previously explored: the fallacy of expert immunity. Our expertise in pharmaceutical quality systems creates cognitive patterns that simultaneously enable rapid, accurate technical judgments while increasing our vulnerability to specific biases.

The very mechanisms that make us effective quality professionals—pattern recognition, schema-based processing, heuristic shortcuts derived from base rate experiences—are the same cognitive tools that generate bias. When I conduct investigations or facilitate risk assessments, my extensive experience with similar events creates expectations and assumptions that can blind me to novel failure modes or unexpected causal relationships. This isn’t a character flaw; it’s an inherent part of how expertise develops and operates.

Valerie’s emphasis on the need for trained facilitators in high-formality QRM activities reflects this reality. External facilitation isn’t just about process management—it’s about introducing cognitive diversity and bias detection capabilities that internal teams, no matter how experienced, cannot provide for themselves. The facilitator serves as a structured intervention against the GI Joe fallacy, embodying the systematic approaches that awareness alone cannot deliver.

From Awareness to Architecture: Building Bias-Resistant Quality Systems

The critical insight from both Valerie’s work and my writing about structured hypothesis formation is that effective bias management requires architectural solutions, not individual willpower. ICH Q9(R1)’s introduction of the “Managing and Minimizing Subjectivity” section represents recognition that regulatory compliance requires systematic approaches to cognitive bias management.

In my post on reducing subjectivity in quality risk management, I identified four strategies that directly address the limitations Valerie highlights about the GI Joe Bias:

Leveraging Knowledge Management: Rather than relying on individual awareness, effective bias management requires systematic capture and application of objective information. When risk assessors can access structured historical data, supplier performance metrics, and process capability studies, they’re less dependent on potentially biased recollections or impressions.
Good Risk Questions: The formulation of risk questions represents a critical intervention point. Well-crafted questions can anchor assessments in specific, measurable terms rather than vague generalizations that invite subjective interpretation. Instead of asking “What are the risks to product quality?”, effective risk questions might ask “What are the potential causes of out-of-specification dissolution results for Product X in the next 6 months based on the last three years of data?”
Cross-Functional Teams: Valerie’s observation that we’re better at spotting biases in others translates directly into team composition strategies. Diverse, cross-functional teams naturally create the external perspective that individual bias recognition cannot provide. The manufacturing engineer, quality analyst, and regulatory specialist bring different cognitive frameworks that can identify blind spots in each other’s reasoning.
Structured Decision-Making Processes: The tools Valerie mentions—PHA, FMEA, Ishikawa, bow-tie analysis—serve as external cognitive scaffolding that guides thinking through systematic pathways rather than relying on intuitive shortcuts that may be biased.

The Formality Framework: When and How to Escalate Bias Management

One of the most valuable aspects of ICH Q9(R1) is its introduction of the formality concept—the idea that different situations require different levels of systematic intervention. Valerie’s article implicitly addresses this by noting that “high formality QRM activities” require trained facilitators. This suggests a graduated approach to bias management that scales intervention intensity with decision importance.

This formality framework needs to include bias management that organizations can use to determine when and how intensively to apply bias mitigation strategies:

Low Formality Situations: Routine decisions with well-understood parameters, limited stakeholders, and reversible outcomes. Basic bias awareness training and standardized checklists may be sufficient.
Medium Formality Situations: Decisions involving moderate complexity, uncertainty, or impact. These require cross-functional input, structured decision tools, and documentation of rationales.
High Formality Situations: Complex, high-stakes decisions with significant uncertainty, multiple conflicting objectives, or diverse stakeholders. These demand external facilitation, systematic bias checks, and formal documentation of how potential biases were addressed.

This framework acknowledges that the GI Joe fallacy is most dangerous in high-formality situations where the stakes are highest and the cognitive demands greatest. It’s precisely in these contexts that our confidence in our ability to overcome bias through awareness becomes most problematic.

The Cultural Dimension: Creating Environments That Support Bias Recognition

Valerie’s emphasis on fostering humility, encouraging teams to acknowledge that “no one is immune to bias, even the most experienced professionals” connects to my observations about building expertise in quality organizations. Creating cultures that can effectively manage subjectivity requires more than tools and processes; it requires psychological safety that allows bias recognition without professional threat.

I’ve noted in past posts that organizations advancing beyond basic awareness levels demonstrate “systematic recognition of cognitive bias risks” with growing understanding that “human judgment limitations can affect risk assessment quality.” However, the transition from awareness to systematic application requires cultural changes that make bias discussion routine rather than threatening.

This cultural dimension becomes particularly important when we consider the ironic processing effects that Valerie references. When organizations create environments where acknowledging bias is seen as admitting incompetence, they inadvertently increase bias through suppression attempts. Teams that must appear confident and decisive may unconsciously avoid bias recognition because it threatens their professional identity.

The solution is creating cultures that frame bias recognition as professional competence rather than limitation. Just as we expect quality professionals to understand statistical process control or regulatory requirements, we should expect them to understand and systematically address their cognitive limitations.

Practical Implementation: Moving Beyond the GI Joe Fallacy

Building on Valerie’s recommendations for structured tools and systematic approaches, here are some specific implementation strategies that organizations can adopt to move beyond bias awareness toward bias management:

Bias Pre-mortems: Before conducting risk assessments, teams explicitly discuss what biases might affect their analysis and establish specific countermeasures. This makes bias consideration routine rather than reactive.
Devil’s Advocate Protocols: Systematic assignment of team members to challenge prevailing assumptions and identify information that contradicts emerging conclusions.
Perspective-Taking Requirements: Formal requirements to consider how different stakeholders (patients, regulators, operators) might view risks differently from the assessment team.
Bias Audit Trails: Documentation requirements that capture not just what decisions were made, but how potential biases were recognized and addressed during the decision-making process.
External Review Requirements: For high-formality decisions, mandatory review by individuals who weren’t involved in the initial assessment and can provide fresh perspectives.

These interventions acknowledge that bias management is not about eliminating human judgment—it’s about scaffolding human judgment with systematic processes that compensate for known cognitive limitations.

The Broader Implications: Subjectivity as Systemic Challenge

Valerie’s analysis of the GI Joe Bias connects to broader themes in my work about the effectiveness paradox and the challenges of building rigorous quality systems in an age of pop psychology. The pharmaceutical industry’s tendency to adopt appealing frameworks without rigorous evaluation extends to bias management strategies. Organizations may implement “bias training” or “awareness programs” that create the illusion of progress while failing to address the systematic changes needed for genuine improvement.

The GI Joe Bias serves as a perfect example of this challenge. It’s tempting to believe that naming the bias—recognizing that awareness isn’t enough—somehow protects us from falling into the awareness trap. But the bias is self-referential: knowing about the GI Joe Bias doesn’t automatically prevent us from succumbing to it when implementing bias management strategies.

This is why Valerie’s emphasis on systematic interventions rather than individual awareness is so crucial. Effective bias management requires changing the decision-making environment, not just the decision-makers’ knowledge. It requires building systems, not slogans.

A Call for Systematic Excellence in Bias Management

Valerie’s exploration of the GI Joe Bias provides a crucial call for advancing pharmaceutical quality management beyond the illusion that awareness equals capability. Her work, combined with ICH Q9(R1)’s explicit recognition of subjectivity challenges, creates an opportunity for the industry to develop more sophisticated approaches to cognitive bias management.

The path forward requires acknowledging that bias management is a core competency for quality professionals, equivalent to understanding analytical method validation or process characterization. It requires systematic approaches that scaffold human judgment rather than attempting to eliminate it. Most importantly, it requires cultures that view bias recognition as professional strength rather than weakness.

As I continue to build frameworks for reducing subjectivity in quality risk management and developing structured approaches to decision-making, Valerie’s insights about the limitations of awareness provide essential grounding. The GI Joe Bias reminds us that knowing is not half the battle—it’s barely the beginning.

The real battle lies in creating pharmaceutical quality systems that systematically compensate for human cognitive limitations while leveraging human expertise and judgment. That battle is won not through individual awareness or good intentions, but through systematic excellence in bias management architecture.

What structured approaches has your organization implemented to move beyond bias awareness toward systematic bias management? Share your experiences and challenges as we work together to advance the maturity of risk management practices in our industry.

Meet Valerie Mulholland

Dr. Valerie Mulholland is transforming how our industry thinks about quality risk management. As CEO and Principal Consultant at GMP Services in Ireland, Valerie brings over 25 years of hands-on experience auditing and consulting across biopharmaceutical, pharmaceutical, medical device, and blood transfusion industries throughout the EU, US, and Mexico.

But what truly sets Valerie apart is her unique combination of practical expertise and cutting-edge research. She recently earned her PhD from TU Dublin’s Pharmaceutical Regulatory Science Team, focusing on “Effective Risk-Based Decision Making in Quality Risk Management”. Her groundbreaking research has produced 13 academic papers, with four publications specifically developed to support ICH’s work—research that’s now incorporated into the official ICH Q9(R1) training materials. This isn’t theoretical work gathering dust on academic shelves; it’s research that’s actively shaping global regulatory guidance.

Why Risk Revolution Deserves Your Attention

The Risk Revolution podcast, co-hosted by Valerie alongside Nuala Calnan (25-year pharmaceutical veteran and Arnold F. Graves Scholar) and Dr. Lori Richter (Director of Risk Management at Ultragenyx with 21+ years industry experience), represents something unique in pharmaceutical podcasting. This isn’t your typical regulatory update show—it’s a monthly masterclass in advancing risk management maturity.

In an industry where staying current isn’t optional—it’s essential for patient safety—Risk Revolution offers the kind of continuing education that actually advances your professional capabilities. These aren’t recycled conference presentations; they’re conversations with the people shaping our industry’s future.

The Minimal Viable Risk Assessment Team

Ineffective risk management and quality systems revolve around superficial risk management. The core issue? Teams designed for compliance as a check-the-box activity rather than cognitive rigor. These gaps create systematic blind spots that no checklist can fix. The solution isn’t more assessors—it’s fewer, more competent ones anchored in science, patient impact, and lived process reality.

Cognitive Foundations of Risk Management Excellence

Core Roles: The Non-Negotiables

1. Process Owner: The Reality Anchor

Not a title. A lived experience. Superficial ownership creates the “unjustified assumptions.” This role requires daily engagement with the process—not just signature authority. Without it, assumptions go unchallenged.

2. ASTM E2500 Molecule Steward: The Patient’s Advocate

Beyond “SME”—the protein whisperer. This role demands provable knowledge of degradation pathways, critical quality attributes (CQAs), and patient impact. Contrast this with generic “subject matter experts” who lack molecule-specific insights. Without this anchor, assessments overlook patient-centric failure modes.

3. Technical System Owner: The Engineer

The value of the Technical System Owner—often the engineer—lies in their unique ability to bridge the worlds of design, operations, and risk control throughout the pharmaceutical lifecycle. Far from being a mere custodian of equipment, the system owner is the architect who understands not just how a system is built, but how it behaves under real-world conditions and how it integrates with the broader manufacturing program

4. Quality: The Cognitive Warper

Forget the auditor—this is your bias disruptor. Quality’s value lies in forcing cross-functional dialogue, challenging tacit assumptions, and documenting debates. When Quality fails to interrogate assumptions, hazards go unidentified. Their real role: Mandate “assumption logs” where every “We’ve always done it this way” must produce data or die.

A Venn diagram with three overlapping blue circles, each representing a different role: "Process Owner: The Reality Anchor," "Molecule Steward: The Patient’s Advocate," and "Technical System Owner: The Engineer." In the center, where all three circles overlap, is a green dashed circle labeled "Quality: Cognitive Warper." Each role has associated bullet points in colored dots:

Process Owner (top left): "Daily Engagement" and "Lived Experience" (blue dots).

Molecule Steward (top right): "Molecular specific insights" and "Patient-centric" (blue dots).

Technical System Owner (bottom): "The How’s" and "Technical understanding" (blue dots).

Additional points for Technical System Owner (bottom right): "Bias disruptor" and "Interrogate assumptions" (green dots).

The diagram visually emphasizes the intersection of these roles in achieving quality through cognitive diversity.

Team Design as Knowledge Preservation

Team design in the context of risk management is fundamentally an act of knowledge preservation, not just an exercise in filling seats or meeting compliance checklists. Every effective risk team is a living repository of the organization’s critical process insights, technical know-how, and nuanced operational experience. When teams are thoughtfully constructed to include individuals with deep, hands-on familiarity—process owners, technical system engineers, molecule stewards, and quality integrators—they collectively safeguard the hard-won lessons and tacit knowledge that are so often lost when people move on or retire. This approach ensures that risk assessments are not just theoretical exercises but are grounded in the practical realities that only those with lived experience can provide.

Combating organizational forgetting requires more than documentation or digital knowledge bases; it demands intentional, cross-functional team design that fosters active knowledge transfer. When a risk team brings together diverse experts who routinely interact, challenge each other’s assumptions, and share context from their respective domains, they create a dynamic environment where critical information is surfaced, scrutinized, and retained. This living dialogue is far more effective than static records, as it allows for the continuous updating and contextualization of knowledge in response to new challenges, regulatory changes, and operational shifts. In this way, team design becomes a strategic defense against the silent erosion of expertise that can leave organizations exposed to avoidable risks.

Ultimately, investing in team design as a knowledge preservation strategy is about building organizational resilience. It means recognizing that the greatest threats often arise not from what is known, but from what is forgotten or never shared. By prioritizing teams that embody both breadth and depth of experience, organizations create a robust safety net—one that catches subtle warning signs, adapts to evolving risks, and ensures that critical knowledge endures beyond any single individual’s tenure. This is how organizations move from reactive problem-solving to proactive risk management, turning collective memory into a competitive advantage and a foundation for sustained quality.

Call to Action: Build the Risk Team

Moving from compliance theater to true protection starts with assembling a team designed for cognitive rigor, knowledge depth and psychological safety.

Start with a Clear Charter, Not a Checklist

An excellent risk team exists to frame, analyse and communicate uncertainty so that the business can make science-based, patient-centred decisions. Assigning authorities and accountabilities is a leadership duty, not an after-thought. Before naming people, write down:

the decisions the team must enable,
the degree of formality those decisions demand, and
the resources (time, data, tools) management will guarantee.

Without this charter, even star performers will default to box-ticking.

Fill Four Core Seats – And Prove Competence

ICH Q9 is blunt: risk work should be done by interdisciplinary teams that include experts from quality, engineering, operations and regulatory affairs. ASTM E2500 translates that into a requirement for documented subject-matter experts (SMEs) who own critical knowledge throughout the lifecycle. Map those expectations onto four non-negotiable roles.

Process Owner – The Reality Anchor: This individual has lived the operation in the last 90 days, not just signed SOPs. They carry the authority to change methods, budgets and training, and enough hands-on credibility to spot when a theoretical control will never work on the line. Authentic owners dismantle assumptions by grounding every risk statement in current shop-floor facts.
Molecule Steward – The Patient’s Advocate: Too often “SME” is shorthand for “the person available.” The molecule steward is different: a scientist who understands how the specific product fails and can translate deviations into patient impact. When temperature drifts two degrees during freeze-drying, the steward can explain whether a monoclonal antibody will aggregate or merely lose a day of shelf life. Without this anchor, the team inevitably under-scores hazards that never appear in a generic FMEA template.
Technical System Owner – The Engineering Interpreter: Equipment does not care about meeting minutes; it obeys physics. The system owner must articulate functional requirements, design limits and integration logic. Where a tool-focused team may obsess over gasket leaks, the system owner points out that a single-loop PLC has no redundancy and that a brief voltage dip could push an entire batch outside critical parameters—a classic case of method over physics.
Quality Integrator – The Bias Disruptor: Quality’s mission is to force cross-functional dialogue and preserve evidence. That means writing assumption logs, challenging confirmation bias and ensuring that dissenting voices are heard. The quality lead also maintains the knowledge repository so future teams are not condemned to repeat forgotten errors.

Secure Knowledge Accessibility, Not Just Possession

A credentialed expert who cannot be reached when the line is down at 2 a.m. is as useful as no expert at all. Conduct a Knowledge Accessibility Index audit before every major assessment.

Embed Psychological Safety to Unlock the Team’s Brainpower

No amount of SOPs compensates for a culture that punishes bad news. Staff speak up only when leaders are approachable, intolerant of blame and transparent about their own fallibility. Leaders must therefore:

Invite dissent early: begin meetings with “What might we be overlooking?”
Model vulnerability: share personal errors and how the system, not individuals, failed.
Reward candor: recognize the engineer who halted production over a questionable trend.

Psychological safety converts silent observers into active risk sensors.

Choose Methods Last, After Understanding the Science

Excellent teams let the problem dictate the tool, not vice versa. They build a failure-tree or block diagram first, then decide whether FMEA, FTA or bow-tie analysis will illuminate the weak spot. If the team defaults to a method because “it’s in the SOP,” stop and reassess. Tool selection is a decision, not a reflex.

Provide Time and Resources Proportionate to Uncertainty

ICH Q9 asks decision-makers to ensure resources match the risk question. Complex, high-uncertainty topics demand longer workshops, more data and external review, while routine changes may only need a rapid check. Resist the urge to shoehorn every assessment into a one-hour meeting because calendars are overloaded.

Institutionalize Learning Loops

Great teams treat every assessment as both analysis and experiment. They:

Track prediction accuracy: did the “medium”-ranked hazard occur?
Compare expected versus actual detectability: were controls as effective as assumed?
Feed insights into updated templates and training so the next team starts smarter.

The loop closes when the knowledge base evolves at the same pace as the plant.

When to Escalate – The Abort-Mission Rule

If a risk scenario involves patient safety, novel technology and the molecule steward is unavailable, stop. The assessment waits until a proper team is in the room. Rushing ahead satisfies schedules, not safety.

Conclusion

Excellence in risk management is rarely about adding headcount; it is about curating brains with complementary lenses and giving them the culture, structure and time to think. Build that environment and the monsters stay on the storyboard, never in the plant.

Assessing the Quality of Our Risk Management Activities

Reducing Subjectivity in Quality Risk Management: Aligning with ICH Q9(R1)

In a previous post, I discussed how overcoming subjectivity in risk management and decision-making requires fostering a culture of quality and excellence. This is an issue that it is important to continue to evaluate and push for additional improvement.

Overcoming Subjectivity in Risk Management and Decision Making Requires a Culture of Quality and Excellence

The revised ICH Q9(R1) guideline, finalized in January 2023, introduces critical updates to Quality Risk Management (QRM) practices, emphasizing the need to address subjectivity, enhance formality, improve risk-based decision-making, and manage product availability risks. These revisions aim to ensure that QRM processes are more science-driven, knowledge-based, and effective in safeguarding product quality and patient safety. Two years later it is important to continue to build on key strategies for reducing subjectivity in QRM and aligning with the updated requirements.

Understanding Subjectivity in QRM

Subjectivity in QRM arises from personal opinions, biases, heuristics, or inconsistent interpretations of risks by stakeholders. This can impact every stage of the QRM process—from hazard identification to risk evaluation and mitigation. The revised ICH Q9(R1) explicitly addresses this issue by introducing a new subsection, “Managing and Minimizing Subjectivity,” which emphasizes that while subjectivity cannot be entirely eliminated, it can be controlled through structured approaches.

The guideline highlights that subjectivity often stems from poorly designed scoring systems, differing perceptions of hazards and risks among stakeholders, and cognitive biases. To mitigate these challenges, organizations must adopt robust strategies that prioritize scientific knowledge and data-driven decision-making.

Strategies to Reduce Subjectivity

Leveraging Knowledge Management

ICH Q9(R1) underscores the importance of knowledge management as a tool to reduce uncertainty and subjectivity in risk assessments. Effective knowledge management involves systematically capturing, organizing, and applying internal and external knowledge to inform QRM activities. This includes maintaining centralized repositories for technical data, fostering real-time information sharing across teams, and learning from past experiences through structured lessons-learned processes.

By integrating knowledge management into QRM, organizations can ensure that decisions are based on comprehensive data rather than subjective estimations. For example, using historical data on process performance or supplier reliability can provide objective insights into potential risks.

To integrate knowledge management (KM) more effectively into quality risk management (QRM), organizations can implement several strategies to ensure decisions are based on comprehensive data rather than subjective estimations:

Establish Robust Knowledge Repositories

Create centralized, easily accessible repositories for storing and organizing historical data, lessons learned, and best practices. These repositories should include:

Process performance data
Supplier reliability metrics
Deviation and CAPA records
Audit findings and inspection observations
Technology transfer documentation

By maintaining these repositories, organizations can quickly access relevant historical information when conducting risk assessments.

Implement Knowledge Mapping

Conduct knowledge mapping exercises to identify key sources of knowledge within the organization. This process helps to:

Locate subject matter experts
Identify critical knowledge assets
Uncover knowledge gaps that may impact risk assessments

Use the resulting knowledge maps to guide risk assessment teams to relevant information and expertise.

Develop Data Analytics Capabilities

Invest in data analytics tools and capabilities to extract meaningful insights from historical data. For example:

Use statistical process control to identify trends in manufacturing performance
Apply machine learning algorithms to predict potential quality issues based on historical patterns
Utilize data visualization tools to present complex risk data in an easily understandable format

These analytics can provide objective, data-driven insights into potential risks and their likelihood of occurrence.

Integrate KM into QRM Processes

Embed KM activities directly into QRM processes to ensure consistent use of available knowledge:

Include a knowledge gathering step at the beginning of risk assessments
Require risk assessment teams to document the sources of knowledge used in their analysis
Implement a formal process for capturing new knowledge generated during risk assessments

This integration helps ensure that all relevant knowledge is considered and that new insights are captured for future use.

Foster a Knowledge-Sharing Culture

Encourage a culture of knowledge sharing and collaboration within the organization:

Implement mentoring programs to facilitate the transfer of tacit knowledge
Establish communities of practice around key risk areas
Recognize and reward employees who contribute valuable knowledge to risk management efforts

By promoting knowledge sharing, organizations can tap into the collective expertise of their workforce to improve risk assessments.

Implementing Structured Risk-Based Decision-Making

The revised guideline introduces a dedicated section on risk-based decision-making, emphasizing the need for structured approaches that consider the complexity, uncertainty, and importance of decisions. Organizations should establish clear criteria for decision-making processes, define acceptable risk tolerance levels, and use evidence-based methods to evaluate options.

Structured decision-making tools can help standardize how risks are assessed and prioritized. Additionally, calibrating expert opinions through formal elicitation techniques can further reduce variability in judgments.

Addressing Cognitive Biases

Cognitive biases—such as overconfidence or anchoring—can distort risk assessments and lead to inconsistent outcomes. To address this, organizations should provide training on recognizing common biases and their impact on decision-making. Encouraging diverse perspectives within risk assessment teams can also help counteract individual biases.

For example, using cross-functional teams ensures that different viewpoints are considered when evaluating risks, leading to more balanced assessments. Regularly reviewing risk assessment outputs for signs of bias or inconsistencies can further enhance objectivity.

Enhancing Formality in QRM

ICH Q9(R1) introduces the concept of a “formality continuum,” which aligns the level of effort and documentation with the complexity and significance of the risk being managed. This approach allows organizations to allocate resources effectively by applying less formal methods to lower-risk issues while reserving rigorous processes for high-risk scenarios.

For instance, routine quality checks may require minimal documentation compared to a comprehensive risk assessment for introducing new manufacturing technologies. By tailoring formality levels appropriately, organizations can ensure consistency while avoiding unnecessary complexity.

Calibrating Expert Opinions

We need to recognize the importance of expert knowledge in QRM activities, but also acknowledges the potential for subjectivity and bias in expert judgments. We need to ensure we:

Implement formal processes for expert opinion elicitation
Use techniques to calibrate expert judgments, especially when estimating probabilities
Provide training on common cognitive biases and their impact on risk assessment
Employ diverse teams to counteract individual biases
Regularly review risk assessment outputs for signs of bias or inconsistencies

Calibration techniques may include:

Structured elicitation protocols that break down complex judgments into more manageable components
Feedback and training to help experts align their subjective probability estimates with actual frequencies of events
Using multiple experts and aggregating their judgments through methods like Cooke’s classical model
Employing facilitation techniques to mitigate groupthink and encourage independent thinking

By calibrating expert opinions, organizations can leverage valuable expertise while minimizing subjectivity in risk assessments.

Utilizing Cooke’s Classical Model

Cooke’s Classical Model is a rigorous method for evaluating and combining expert judgments to quantify uncertainty. Here are the key steps for using the Classical Model to evaluate expert judgment:

Select and calibrate experts:

Choose 5-10 experts in the relevant field
Have experts assess uncertain quantities (“calibration questions”) for which true values are known or will be known soon
These calibration questions should be from the experts’ domain of expertise

Elicit expert assessments:

Have experts provide probabilistic assessments (usually 5%, 50%, and 95% quantiles) for both calibration questions and questions of interest
Document experts’ reasoning and rationales

Score expert performance:

Evaluate experts on two measures:
a) Statistical accuracy: How well their probabilistic assessments match the true values of calibration questions
b) Informativeness: How precise and focused their uncertainty ranges are

Calculate performance-based weights:

Derive weights for each expert based on their statistical accuracy and informativeness scores
Experts performing poorly on calibration questions receive little or no weight

Combine expert assessments:

Use the performance-based weights to aggregate experts’ judgments on the questions of interest
This creates a “Decision Maker” combining the experts’ assessments

Validate the combined assessment:

Evaluate the performance of the weighted combination (“Decision Maker”) using the same scoring as for individual experts
Compare to equal-weight combination and best-performing individual experts

Conduct robustness checks:

Perform cross-validation by using subsets of calibration questions to form weights
Assess how well performance on calibration questions predicts performance on questions of interest

The Classical Model aims to create an optimal aggregate assessment that outperforms both equal-weight combinations and individual experts. By using objective performance measures from calibration questions, it provides a scientifically defensible method for evaluating and synthesizing expert judgment under uncertainty.

Using Data to Support Decisions

ICH Q9(R1) emphasizes the importance of basing risk management decisions on scientific knowledge and data. The guideline encourages organizations to:

Develop robust knowledge management systems to capture and maintain product and process knowledge
Create standardized repositories for technical data and information
Implement systems to collect and convert data into usable knowledge
Gather and analyze relevant data to support risk-based decisions
Use quantitative methods where feasible, such as statistical models or predictive analytics

Specific approaches for using data in QRM may include:

Analyzing historical data on process performance, deviations, and quality issues to inform risk assessments
Employing statistical process control and process capability analysis to evaluate and monitor risks
Utilizing data mining and machine learning techniques to identify patterns and potential risks in large datasets
Implementing real-time data monitoring systems to enable proactive risk management
Conducting formal data quality assessments to ensure decisions are based on reliable information

Digitalization and emerging technologies can support data-driven decision making, but remember that validation requirements for these technologies should not be overlooked.

Improving Risk Assessment Tools

The design of risk assessment tools plays a critical role in minimizing subjectivity. Tools with well-defined scoring criteria and clear guidance on interpreting results can reduce variability in how risks are evaluated. For example, using quantitative methods where feasible—such as statistical models or predictive analytics—can provide more objective insights compared to qualitative scoring systems.

Organizations should also validate their tools periodically to ensure they remain fit-for-purpose and aligned with current regulatory expectations.

Leverage Good Risk Questions

A well-formulated risk question can significantly help reduce subjectivity in quality risk management (QRM) activities. Here’s how a good risk question contributes to reducing subjectivity:

Clarity and Focus

A good risk question provides clarity and focus for the risk assessment process. By clearly defining the scope and context of the risk being evaluated, it helps align all participants on what specifically needs to be assessed. This alignment reduces the potential for individual interpretations and subjective assumptions about the risk scenario.

Specific and Measurable Terms

Effective risk questions use specific and measurable terms rather than vague or ambiguous language. For example, instead of asking “What are the risks to product quality?”, a better question might be “What are the potential causes of out-of-specification dissolution results for Product X in the next 6 months?”. The specificity in the latter question helps anchor the assessment in objective, measurable criteria.

Factual Basis

A well-crafted risk question encourages the use of factual information and data rather than opinions or guesses. It should prompt the risk assessment team to seek out relevant data, historical information, and scientific knowledge to inform their evaluation. This focus on facts and evidence helps minimize the influence of personal biases and subjective judgments.

Standardized Approach

Using a consistent format for risk questions across different assessments promotes a standardized approach to risk identification and analysis. This consistency reduces variability in how risks are framed and evaluated, thereby decreasing the potential for subjective interpretations.

Objective Criteria

Good risk questions often incorporate or imply objective criteria for risk evaluation. For instance, a question like “What factors could lead to a deviation from the acceptable range of 5-10% for impurity Y?” sets clear, objective parameters for the assessment, reducing the room for subjective interpretation of what constitutes a significant risk.

Promotes Structured Thinking

Well-formulated risk questions encourage structured thinking about potential hazards, their causes, and consequences. This structured approach helps assessors focus on objective factors and causal relationships rather than relying on gut feelings or personal opinions.

Facilitates Knowledge Utilization

A good risk question should prompt the assessment team to utilize available knowledge effectively. It encourages the team to draw upon relevant data, past experiences, and scientific understanding, thereby grounding the assessment in objective information rather than subjective impressions.

By crafting risk questions that embody these characteristics, QRM practitioners can significantly reduce the subjectivity in risk assessments, leading to more reliable, consistent, and scientifically sound risk management decisions.

Fostering a Culture of Continuous Improvement

Reducing subjectivity in QRM is an ongoing process that requires a commitment to continuous improvement. Organizations should regularly review their QRM practices to identify areas for enhancement and incorporate feedback from stakeholders. Investing in training programs that build competencies in risk assessment methodologies and decision-making frameworks is essential for sustaining progress.

Moreover, fostering a culture that values transparency, collaboration, and accountability can empower teams to address subjectivity proactively. Encouraging open discussions about uncertainties or disagreements during risk assessments can lead to more robust outcomes.

Conclusion

The revisions introduced in ICH Q9(R1) represent a significant step forward in addressing long-standing challenges associated with subjectivity in QRM. By leveraging knowledge management, implementing structured decision-making processes, addressing cognitive biases, enhancing formality levels appropriately, and improving risk assessment tools, organizations can align their practices with the updated guidelines while ensuring more reliable and science-based outcomes.

It has been two years, it is long past time be be addressing these in your risk management process and quality system.

Ultimately, reducing subjectivity not only strengthens compliance with regulatory expectations but also enhances the quality of pharmaceutical products and safeguards patient safety—a goal that lies at the heart of effective Quality Risk Management.