Assessing the Quality of Our Risk Management Activities

Twenty years on, risk management in the pharmaceutical world continues to be challenging. Ensure that risk assessments are systematic, structured, and based on scientific knowledge. A large part of the ICH Q9(R1) revision was written to address continued struggles with subjectivity, formality, and decision-making. And quite frankly, it’s clear to me that we, as an industry, are still working to absorb those messages these last two years.

A big challenge is that we struggle to measure the effectiveness of our risk assessments. Quite frankly, this is a great place for a rubric.

Luckily, we have a good tool out there to adopt: the Risk Analysis Quality Test (RAQT1.0), developed by the Society for Risk Analysis (SRA). This comprehensive framework is designed to evaluate and improve the quality of risk assessments. We can apply this tool to meet the requirements of the International Conference on Harmonisation (ICH) Q9, which outlines quality risk management principles for the pharmaceutical industry. From that, we can drive continued improvement in our risk management activities.

Components of RAQT1.0

The Risk Analysis Quality Test consists of 76 questions organized into 15 categories:

Framing the Analysis and Its Interface with Decision Making
Capturing the Risk Generating Process (RGP)
Communication
Stakeholder Involvement
Assumptions and Scope Boundary Issues
Proactive Creation of Alternative Courses of Action
Basis of Knowledge
Data Limitations
Analysis Limitations
Uncertainty
Consideration of Alternative Analysis Approaches
Robustness and Resilience of Action Strategies
Model and Analysis Validation and Documentation
Reporting
Budget and Schedule Adequacy

Application to ICH Q9 Requirements

ICH Q9 emphasizes the importance of a systematic and structured risk assessment process. The RAQT can be used to ensure that risk assessments are thorough and meet quality standards. For example, Category G (Basis of Knowledge) and Category H (Data Limitations) help in evaluating the scientific basis and data quality of the risk assessment, aligning with ICH Q9’s requirement for using available knowledge and data.

The RAQT’s Category B (Capturing the Risk Generating Process) and Category C (Communication) can help in identifying and communicating risks effectively. This aligns with ICH Q9’s requirement to identify potential risks based on scientific knowledge and understanding of the process.

Categories such as Category I (Analysis Limitations) and Category J (Uncertainty) in the RAQT help in analyzing the risks and addressing uncertainties, which is a key aspect of ICH Q9. These categories ensure that the analysis is robust and considers all relevant factors.

The RAQT’s Category A (Framing the Analysis and Its Interface with Decision Making) and Category F (Proactive Creation of Alternative Courses of Action) are crucial for evaluating risks and developing mitigation strategies. This aligns with ICH Q9’s requirement to evaluate risks and determine the need for risk reduction.

Categories like Category L (Robustness and Resilience of Action Strategies) and Category M (Model and Analysis Validation and Documentation) in the RAQT help in ensuring that the risk control measures are robust and well-documented. This is consistent with ICH Q9’s emphasis on implementing and reviewing controls.

Category D (Stakeholder Involvement) of the RAQT ensures that stakeholders are engaged in the risk management process, which is a requirement under ICH Q9 for effective communication and collaboration.

The RAQT can be applied both retrospectively and prospectively, allowing for the evaluation of past risk assessments and the planning of future ones. This aligns with ICH Q9’s requirement for periodic review and continuous improvement of the risk management process.

Creating a Rubric

To make this actionable we need a tool, a rubric, to allow folks to evaluate what goods look like. I would insert this tool into the quality oversite of risk management.

Category A: Framing the Analysis and Its Interface With Decision Making

Criteria	Excellent (4)	Good (3)	Fair (2)	Poor (1)
Problem Definition	Clearly and comprehensively defines the problem, including all relevant aspects and stakeholders	Adequately defines the problem with most relevant aspects considered	Partially defines the problem with some key aspects missing	Poorly defines the problem or misses critical aspects
Analytical Approach	Selects and justifies an optimal analytical approach, demonstrating deep understanding of methodologies	Chooses an appropriate analytical approach with reasonable justification	Selects a somewhat relevant approach with limited justification	Chooses an inappropriate approach or provides no justification
Data Collection and Management	Thoroughly identifies all necessary data sources and outlines a comprehensive data management plan	Identifies most relevant data sources and provides a adequate data management plan	Identifies some relevant data sources and offers a basic data management plan	Fails to identify key data sources or lacks a coherent data management plan
Stakeholder Identification	Comprehensively identifies all relevant stakeholders and their interests	Identifies most key stakeholders and their primary interests	Identifies some stakeholders but misses important ones or their interests	Fails to identify major stakeholders or their interests
Decision-Making Context	Provides a thorough analysis of the decision-making context, including constraints and opportunities	Adequately describes the decision-making context with most key factors considered	Partially describes the decision-making context, missing some important factors	Poorly describes or misunderstands the decision-making context
Alignment with Organizational Goals	Demonstrates perfect alignment between the analysis and broader organizational objectives	Shows good alignment with organizational goals, with minor gaps	Partially aligns with organizational goals, with significant gaps	Fails to align with or contradicts organizational goals
Communication Strategy	Develops a comprehensive strategy for communicating results to all relevant decision-makers	Outlines a good communication strategy covering most key decision-makers	Provides a basic communication plan with some gaps	Lacks a clear strategy for communicating results to decision-makers

This rubric provides a framework for assessing the quality of work in framing an analysis and its interface with decision-making. It covers key aspects such as problem definition, analytical approach, data management, stakeholder consideration, decision-making context, alignment with organizational goals, and communication strategy. Each criterion is evaluated on a scale from 1 (Poor) to 4 (Excellent), allowing for nuanced assessment of performance in each area.

To use this rubric effectively:

Adjust the criteria and descriptions as needed to fit your specific context or requirements.
Ensure that the expectations for each level (Excellent, Good, Fair, Poor) are clear and distinguishable.

My next steps will be to add specific examples or indicators for each level to provide more guidance to both assessors and those being assessed.

I also may, depending on internal needs, want to assign different weights to each criterion based on their relative importance in your specific context. In this case I think each ends up being pretty similar.

I would then go and add the other sections. For example, here is category B with some possible weighting.

Category B: Capturing the Risk Generating Process (RGP)

Component	Weight Factor	Excellent	Satisfactory	Needs Improvement	Poor
B1. Comprehensiveness	4	The analysis includes: i) A structured taxonomy of hazards/events demonstrating comprehensiveness ii) Each scenario spelled out with causes and types of change iii) Explicit addressing of potential “Black Swan” events iv) Clear description of implications of such events for risk management	The analysis includes 3 out of 4 elements from the Excellent criteria, with minor gaps that do not significantly impact understanding	The analysis includes only 2 out of 4 elements from the Excellent criteria, or has significant gaps in comprehensiveness	The analysis includes 1 or fewer elements from the Excellent criteria, severely lacking in comprehensiveness
B2. Basic Structure of RGP	2	Clearly identifies and accounts for the basic structure of the RGP (e.g. linear, chaotic, complex adaptive) AND Uses appropriate mathematical structures (e.g. linear, quadratic, exponential) that match the RGP structure	Identifies the basic structure of the RGP BUT does not fully align mathematical structures with the RGP	Attempts to identify the RGP structure but does so incorrectly or incompletely OR Uses mathematical structures that do not align with the RGP	Does not identify or account for the basic structure of the RGP
B3. Complexity of RGP	3	Lists all important causal and associative links in the RGP AND Demonstrates how each link is accounted for in the analysis	Lists most important causal and associative links in the RGP AND Demonstrates how most links are accounted for in the analysis	Lists some causal and associative links but misses key elements OR Does not adequately demonstrate how links are accounted for in the analysis	Does not list causal and associative links or account for them in the analysis
B4. Early Warning Detection	3	Includes a clear process for detecting early warnings of potential surprising risk aspects, beyond just concrete events	Includes a process for detecting early warnings, but it may be limited in scope or not fully developed	Mentions the need for early warning detection but does not provide a clear process	Does not address early warning detection
B5. System Changes	2	Fully considers the possibility of system changes AND Establishes adequate mechanisms to detect those changes	Considers the possibility of system changes BUT mechanisms to detect changes are not fully developed	Mentions the possibility of system changes but does not adequately consider or establish detection mechanisms	Does not consider or address the possibility of system changes

I definitely need to go back and add more around structure requirements. The SRA RAQT tool needs some more interpretation here.

Category C: Risk Communication

Component	Weight Factor	Excellent	Satisfactory	Needs Improvement	Poor
C1. Integration of Communication into Risk Analysis	3	Communication is fully integrated into the risk analysis following established norms). All aspects of the methodology are clearly addressed including context establishment, risk assessment (identification, analysis, evaluation), and risk treatment. There is clear evidence of pre-assessment, management, appraisal, characterization and evaluation. Knowledge about the risk is thoroughly categorized.	Communication is integrated into the risk analysis following most aspects of established norms. Most key elements of methodologies like ISO 31000 or IRGC are addressed, but some minor aspects may be missing or unclear. Knowledge about the risk is categorized, but may lack some detail.	Communication is partially integrated into the risk analysis, but significant aspects of established norms are missing. Only some elements of methodologies like ISO 31000 or IRGC are addressed. Knowledge categorization about the risk is incomplete or unclear.	There is little to no evidence of communication being integrated into the risk analysis following established norms. Methodologies like ISO 31000 or IRGC are not followed. Knowledge about the risk is not categorized.
C2. Adequacy of Risk Communication	3	All considerations for effective risk communication have been applied to ensure adequacy between analysts and decision makers, analysts and other stakeholders, and decision makers and stakeholders. There is clear evidence that all parties agree the communication is adequate.	Most considerations for effective risk communication have been applied. Communication appears adequate between most parties, but there may be minor gaps or areas where agreement on adequacy is not explicitly stated.	Some considerations for effective risk communication have been applied, but there are significant gaps. Communication adequacy is questionable between one or more sets of parties. There is limited evidence of agreement on communication adequacy.	Few to no considerations for effective risk communication have been applied. There is no evidence of adequate communication between analysts, decision makers, and stakeholders. There is no indication of agreement on communication adequacy.

Category D: Stakeholder Involvement

Criteria	Weight	Excellent (4)	Satisfactory (3)	Needs Improvement (2)	Poor (1)
Stakeholder Identification	4	All relevant stakeholders are systematically and comprehensively identified	Most relevant stakeholders are identified, with minor omissions	Some relevant stakeholders are identified, but significant groups are missed	Few or no relevant stakeholders are identified
Stakeholder Consultation	3	All identified stakeholders are thoroughly consulted, with their perceptions and concerns fully considered	Most identified stakeholders are consulted, with their main concerns considered	Some stakeholders are consulted, but consultation is limited in scope or depth	Few or no stakeholders are consulted
Stakeholder Engagement	3	Stakeholders are actively engaged throughout the entire risk management process, including problem framing, decision-making, and implementation	Stakeholders are engaged in most key stages of the risk management process	Stakeholders are engaged in some aspects of the risk management process, but engagement is inconsistent	Stakeholders are minimally engaged or not engaged at all in the risk management process
Effectiveness of Involvement	2	All stakeholders would agree that they were effectively consulted and engaged	Most stakeholders would agree that they were adequately consulted and engaged	Some stakeholders may feel their involvement was insufficient or ineffective	Most stakeholders would likely feel their involvement was inadequate or ineffective

Category E: Assumptions and Scope Boundary Issues

Criterion	Weight	Excellent (4)	Satisfactory (3)	Needs Improvement (2)	Poor (1)
E1. Important assumptions and implications listed	4	All important assumptions and their implications for risk management are systematically listed in clear language understandable to decision makers. Comprehensive and well-organized.	Most important assumptions and implications are listed in language generally clear to decision makers. Some minor omissions or lack of clarity.	Some important assumptions and implications are listed, but significant gaps exist. Language is not always clear to decision makers.	Few or no important assumptions and implications are listed. Language is unclear or incomprehensible to decision makers.
E2. Risks of assumption deviations evaluated	3	Risks of all significant assumptions deviating from the actual Risk Generating Process are thoroughly evaluated. Consequences and implications are clearly communicated to decision makers.	Most risks of significant assumption deviations are evaluated. Consequences and implications are generally communicated to decision makers, with minor gaps.	Some risks of assumption deviations are evaluated, but significant gaps exist. Communication to decision makers is incomplete or unclear.	Few or no risks of assumption deviations are evaluated. Little to no communication of consequences and implications to decision makers.
E3. Scope boundary issues and implications listed	3	All important scope boundary issues and their implications for risk management are systematically listed in clear language understandable to decision makers. Comprehensive and well-organized.	Most important scope boundary issues and implications are listed in language generally clear to decision makers. Some minor omissions or lack of clarity.	Some important scope boundary issues and implications are listed, but significant gaps exist. Language is not always clear to decision makers.	Few or no important scope boundary issues and implications are listed. Language is unclear or incomprehensible to decision makers.

Category F: Proactive Creation of Alternative Courses of Action

Criteria	Weight	Excellent (4)	Satisfactory (3)	Needs Improvement (2)	Poor (1)
Systematic generation of alternatives	4	A comprehensive and structured process is used to systematically generate a wide range of alternative courses of action, going well beyond initially considered options	A deliberate process is used to generate multiple alternative courses of action beyond those initially considered	Some effort is made to generate alternatives, but the process is not systematic or comprehensive	Little to no effort is made to generate alternatives beyond those initially considered
Goal-focused creation	3	All generated alternatives are clearly aligned with and directly address the stated goals of the analysis	Most generated alternatives align with the stated goals of the analysis	Some generated alternatives align with the goals, but others seem tangential or unrelated	Generated alternatives (if any) do not align with or address the stated goals
Consideration of robust/resilient options	3	Multiple robust and resilient alternatives are developed to address various uncertainty scenarios	At least one robust or resilient alternative is developed to address uncertainty	Robustness and resilience are considered, but not fully incorporated into alternatives	Robustness and resilience are not considered in alternative generation
Examination of unintended consequences	2	Thorough examination of potential unintended consequences for each alternative, including action-reaction spirals	Some examination of potential unintended consequences for most alternatives	Limited examination of unintended consequences for some alternatives	No consideration of potential unintended consequences
Documentation of alternative creation process	1	The process of alternative generation is fully documented, including rationale for each alternative	The process of alternative generation is mostly documented	The process of alternative generation is partially documented	The process of alternative generation is not documented

Category G: Basis of Knowledge

Criterion	Weight	Excellent (4)	Satisfactory (3)	Needs Improvement (2)	Poor (1)
G1. Characterization of knowledge basis	4	All inputs are clearly characterized (empirical, expert elicitation, testing, modeling, etc.). Distinctions between broadly accepted and novel analyses are explicitly stated.	Most inputs are characterized, with some minor omissions. Distinctions between accepted and novel analyses are mostly clear.	Some inputs are characterized, but significant gaps exist. Limited distinction between accepted and novel analyses.	Little to no characterization of knowledge basis. No distinction between accepted and novel analyses.
G2. Strength of knowledge adequacy	3	Strength of knowledge is thoroughly characterized in terms of its adequacy to support risk management decisions. Limitations are clearly articulated.	Strength of knowledge is mostly characterized, with some minor gaps in relating to decision support adequacy.	Limited characterization of knowledge strength. Unclear how it relates to decision support adequacy.	No characterization of knowledge strength or its adequacy for decision support.
G3. Communication of knowledge limitations	4	All knowledge limitations and their implications for risk management are clearly communicated to decision makers in understandable language.	Most knowledge limitations and implications are communicated, with minor clarity issues.	Some knowledge limitations are communicated, but significant gaps exist in clarity or completeness.	Knowledge limitations are not communicated or are presented in a way decision makers cannot understand.
G4. Consideration of surprises and unforeseen events	3	Thorough consideration of potential surprises and unforeseen events (Black Swans). Their importance is clearly articulated.	Consideration of surprises and unforeseen events is present, with some minor gaps in articulating their importance.	Limited consideration of surprises and unforeseen events. Their importance is not clearly articulated.	No consideration of surprises or unforeseen events.
G5. Conflicting expert opinions	2	All conflicting expert opinions are systematically considered and reported to decision makers as a source of uncertainty.	Most conflicting expert opinions are considered and reported, with minor omissions.	Some conflicting expert opinions are considered, but significant gaps exist in reporting or consideration.	Conflicting expert opinions are not considered or reported.
G6. Consideration of unconsidered knowledge	2	Explicit measures are implemented to check for knowledge outside the analysis group (e.g., independent review).	Some measures are in place to check for outside knowledge, but they may not be comprehensive.	Limited consideration of knowledge outside the analysis group. No formal measures in place.	No consideration of knowledge outside the analysis group.
G7. Consideration of disregarded low-probability events	1	Explicit measures are implemented to check for events disregarded due to low probabilities based on critical assumptions.	Some consideration of low-probability events, but measures may not be comprehensive.	Limited consideration of low-probability events. No formal measures in place.	No consideration of events disregarded due to low probabilities.

This rubric, once done, is a tool to guide assessment and provide feedback. It should be flexible enough to accommodate unique aspects of individual work while maintaining consistent standards across evaluations. I’d embed it in the quality approval step.

Validating Manufacturing Process Closure for Biotech Utilizing Single-Use Systems (SUS)

Maintaining process closure is crucial for ensuring product quality and safety in biotechnology manufacturing, especially when using single-use systems (SUS). This approach is an integral part of the contamination control strategy (CCS). To validate process closure in SUS-based biotech manufacturing, a comprehensive method is necessary, incorporating:

Risk assessment
Thorough testing
Ongoing monitoring

By employing risk analysis tools such as Hazard Analysis and Critical Control Points (HACCP) and Failure Mode and Effects Analysis (FMEA), manufacturers can identify potential weaknesses in their processes. Additionally, addressing all four layers of protection helps ensure process integrity and product safety. This risk-based approach to process closure validation is essential for maintaining the high standards required in biotechnology manufacturing, including meeting Annex 1.

Understanding Process Closure

Process closure refers to the isolation of the manufacturing process from the external environment to prevent contamination. In biotech, this is particularly crucial due to the sensitivity of biological products and the potential for microbial contamination.

The Four Layers of Protection

Throughout this process it is important to apply the four layers of protection that form the foundation of a robust contamination control strategy:

Process: The inherent ability of the process to prevent or control contamination
Equipment: The design and functionality of equipment to maintain closure
Operating Procedures: The practices and protocols followed by personnel
Production Environment: The controlled environment surrounding the process

I was discussing this with some colleagues this week (preparing for some risk assessments) and I was reminded that we really should put the Patient in at the center, the zero. Truer words have never been spoken as the patient truly is our zeroth law, the fundamental principle of the GxPs.

Key Steps for Validating Process Closure

Risk Assessment

Start with a comprehensive risk assessment using tools such as HACCP (Hazard Analysis and Critical Control Points) and FMEA (Failure Mode and Effects Analysis). It is important to remember this is not a one or another, but a multi-tiered approach where you first determine the hazards through the HACCP and then drill down into failures through an FMEA.

HACCP Approach

In the HACCP we will apply a systematic, preventative approach to identify hazards in the process with the aim to produce a documented plan to control these scenarios.

a) Conduct a hazard analysis
b) Identify Critical Control Points (CCPs)
c) Establish critical limits
d) Implement monitoring procedures
e) Define corrective actions
f) Establish verification procedures
g) Maintain documentation and records

FMEA Considerations

In the FMEA we will look for ways the process fails, focusing on the SUS components. We will evaluate failures at each level of control (process, equipment, operating procedure and environment).

Identify potential failure modes in the SUS components
Assess the severity, occurrence, and detectability of each failure mode
Calculate Risk Priority Numbers (RPN) to prioritize risks

Verification

Utilizing these risk assessments, define the user requirements specification (URS) for the SUS, focusing on critical aspects that could impact product quality and patient safety. This should include:

Process requirements (e.g. working volumes, flow rates, pressure ranges)
Material compatibility requirements
Sterility/bioburden control requirements
Leachables/extractables requirements
Integrity testing requirements
Connectivity and interface requirements

Following the ASTM E2500 approach, when we conduct the design review of the proposed SUS configuration, to evaluate how well it meets the URS, we want to ensure we cover:

Overall system design and component selection
Materials of construction
Sterilization/sanitization approach
Integrity assurance measures
Sampling and monitoring capabilities
Automation and control strategy

Circle back to the HACCP and FMEA to ensure they appropriately cover critical aspects like:

Loss of sterility/integrity
Leachables/extractables introduction
Bioburden control failures
Cross-contamination risks
Process parameter deviations

These risk assessments will define critical control parameters and acceptance criteria based on the risk assessment. These will form the basis for verification testing. We will through our verification plan have an appropriate approach to:

Verify proper installation of SUS components
Check integrity of connections and seals
Confirm correct placement of sensors and monitoring devices
Document as-built system configuration
Test system integrity under various operating conditions
Perform leak tests on connections and seals
Validate sterilization processes for SUS components
Verify functionality of critical sensors and control
Run simulated production cycles
Monitor for contamination using sensitive detection methods
Verify maintenance of sterility throughout the process
Assess product quality attributes

The verification strategy will leverage a variety of supplier documentation and internal testing.

Closure Analysis Risk Assessment (CLARA)

Acceptance and release will be to perform a detailed CLARA to:

Identify all potential points of contamination ingress
Assess the effectiveness of closure mechanisms
Evaluate the robustness of aseptic connections
Determine the impact of manual interventions on system closure

On Going Use

Coming out of our HACCP we will have a monitoring and verification plan, this will include some important aspects based on our CCPs.

Integrity Testing
- Implement routine integrity testing protocols for SUS components
- Utilize methods such as pressure decay tests or helium leak detection
- Establish acceptance criteria for integrity tests
Environmental Monitoring
- Develop a comprehensive environmental monitoring program
- Include viable and non-viable particle monitoring
- Establish alert and action limits for environmental contaminants
Operator Training and Qualification
- Develop detailed SOPs for SUS handling and assembly
- Implement a rigorous training program for operators
- Qualify operators through practical assessments
Change Control and Continuous Improvement
- Establish a robust change control process for any modifications to the SUS or process
- Regularly review and update risk assessments based on new data or changes
- Implement a continuous improvement program to enhance process closure

Leveraging the Four Layers of Protection

Throughout the validation process, ensure that each layer of protection is addressed:

Process:
- Optimize process parameters to minimize contamination risks
- Implement in-process controls to detect deviations
Equipment:
- Validate the design and functionality of SUS components
- Ensure proper integration of SUS with existing equipment
Operating Procedures:
- Develop and validate aseptic techniques for SUS handling
- Implement procedures for system assembly and disassembly
Production Environment:
- Qualify the cleanroom environment
- Validate HVAC systems and air filtration

Remember that validation is an ongoing process. Regular reviews, updates to risk assessments, and incorporation of new technologies and best practices are essential for maintaining a state of control in biotech manufacturing using single-use systems.

Connected to the Contamination Control Strategy

Closed systems are a key element of the overall contamination control strategy with closed processing and closed systems now accepted as the most effective contamination control risk mitigation strategy. I might not be able to manufacture in the woods yet, but darn if I won’t keep trying.

They serve as a primary barrier to prevent contamination from the manufacturing environment by helping to mitigate the risk of contamination by isolating the product from the surrounding environment. Closed systems are the key protective measure to prevent contamination from the manufacturing environment and cross-contamination from neighboring operations.

The risk assessments leveraged during the implementation of closed systems are a crucial part of developing an effective CCS and will communicate the (ideally) robust methods used to protect products from environmental contamination and cross-contamination. This is tied into the facility design, environmental controls, risk assessments, and overall manufacturing strategies, which are the key components of a comprehensive CCS.

Risk Assessments as part of Design and Verification

Facility design and manufacturing processes are complex, multi-stage operations, fraught with difficulty. Ensuring the facility meets Good Manufacturing Practice (GMP) standards and other regulatory requirements is a major challenge. The complex regulations around biomanufacturing facilities require careful planning and documentation from the earliest design stages.

Which is why consensus standards like ASTM E2500 exist.

Central to these approaches are risk assessment, to which there are three primary components:

An understanding of the uncertainties in the design (which includes materials, processing, equipment, personnel, environment, detection systems, feedback control)
An identification of the hazards and failure mechanisms
An estimation of the risks associated with each hazard and failure

Folks often get tied up on what tool to use. Frankly, this is a phase approach. We start with a PHA for design, an FMEA for verification and a HACCP/Layers of Control Analysis for Acceptance. Throughout we use a bow-tie for communication.

Aspect	Bow-Tie	PHA (Preliminary Hazard Analysis)	FMEA (Failure Mode and Effects Analysis)	HACCP (Hazard Analysis and Critical Control Points)
Primary Focus	Visualizing risk pathways	Early hazard identification	Potential failure modes	Systematically identify, evaluate, and control hazards that could compromise product safety
Timing in Process	Any stage	Early development	Any stage, often design	Throughout production
Approach	Combines causes and consequences	Top-down	Bottom-up	Systematic prevention
Complexity	Moderate	Low to moderate	High	Moderate
Visual Representation	Central event with causes and consequences	Tabular format	Tabular format	Flow diagram with CCPs
Risk Quantification	Can include, not required	Basic risk estimation	Risk Priority Number (RPN)	Not typically quantified
Regulatory Alignment	Less common in pharma	Aligns with ISO 14971	Widely accepted in pharma	Less common in pharma
Critical Points	Identifies barriers	Does not specify	Identifies critical failure modes	Identifies Critical Control Points (CCPs)
Scope	Specific hazardous event	System-level hazards	Component or process-level failures	Process-specific hazards
Team Requirements	Cross-functional	Less detailed knowledge needed	Detailed system knowledge	Food safety expertise
Ongoing Management	Can be used for monitoring	Often updated periodically	Regularly updated	Continuous monitoring of CCPs
Output	Visual risk scenario	List of hazards and initial risk levels	Prioritized list of failure modes	HACCP plan with CCPs
Typical Use in Pharma	Risk communication	Early risk identification	Detailed risk analysis	Product Safety/Contamination Control

At BOSCON this year I’ll be talking about this fascinating detail, perhaps too much detail.

Conducting A Hazard and Operability Study (HAZOP)

A Hazard and Operability Study (HAZOP) is a structured and systematic examination of a complex planned or existing process or operation to identify and evaluate problems that may represent risks to product, personnel or equipment. The primary goal of a HAZOP is to ensure that risks are managed effectively by identifying potential hazards and operability problems and developing appropriate mitigation strategies.

Why Use HAZOP?

Biotech facilities involve intricate processes that can be prone to various risks, including contamination, equipment failure, and process deviations. Implementing a HAZOP can:

Risk Identification and Mitigation: HAZOPs help identify potential hazards associated with biotech processes, such as contamination risks, equipment malfunctions, and deviations from standard operating procedures. By identifying these risks, facilities can implement mitigation strategies to prevent accidents and ensure safety.
Process Optimization: Through the systematic analysis of processes, HAZOPs can identify inefficiencies and areas for improvement, leading to optimized operations and enhanced productivity.

Part of a Continuum of Risk Tools

A HAZOP (Hazard and Operability) study differs from other risk assessment methods in a few key ways:

Systematic examination of process deviations: HAZOP uses a very structured approach of examining potential deviations from the intended design and operation of a process, using guidewords like “more”, “less”, “no”, “reverse”, etc. This systematic approach helps identify hazards that may be missed by other methods.
Focus on operability issues: The HAZOP examines operability problems that could impact process efficiency or product quality.
Node-by-node analysis: The process is broken down into nodes or sections that are analyzed individually, allowing for very thorough examination.
Qualitative analysis: Unlike quantitative risk assessment methods, HAZOP is primarily qualitative, focusing on identifying potential hazards rather than quantifying risk levels. HAZOPs do not typically assign numerical scores or rankings to risks.
Consideration of causes and consequences: For each deviation, the team examines possible causes, consequences, and existing safeguards before recommending additional actions.
Applicable to complex processes: The structured approach makes HAZOP well-suited for analyzing complex processes with many variables and potential interactions.

Method	Description	Strengths	Limitations
HAZOP (Hazard and Operability Study)	Systematic examination of process/operation to identify potential hazards and operability problems	– Very thorough and structured approach – Examines deviations from design intent – Team-based	– Time consuming – Primarily qualitative
FMEA (Failure Mode and Effects Analysis)	Systematic method to identify potential failure modes and their effects	– Quantitative risk prioritization – Proactive approach – Can be used on products and processes	– Does not consider combinations of failures – Can be subjective
HACCP (Hazard Analysis and Critical Control Points)	Systematic approach to food safety hazards	– Focus on prevention – Identifies critical control points	– Requires prerequisite programs in place
PHA (Preliminary Hazard Analysis)	Early stage hazard identification technique	– Can be used early in design process – Relatively quick to perform – Identifies major hazards	– Not very detailed – Qualitative only – May miss some hazards
Bow-Tie Analysis	Combines fault tree and event tree analysis	– Visual representation of risk pathways – Shows preventive and mitigative controls – Good communication tool	– Does not show detailed failure logic – Can oversimplify complex scenarios – Time consuming for multiple hazards

Key differences:

HAZOP focuses on deviations from design intent, while FMEA looks at potential failure modes
HACCP is specific to identify hazards and is commonly used in food safety, while the others are more general risk assessment tools
PHA is used early in design, while the others are typically used on existing systems
Bow-Tie provides a visual risk pathway, while the others use more tabular formats
FMEA and HAZOP tend to be the most thorough and time-intensive methods

The choice of method depends on the specific application, stage of design, and level of detail required. Often a combination of methods may be used.

Instructions for Conducting a HAZOP

Preparation

Assemble a multidisciplinary team comprising appropriate experts
Define the scope of the HAZOP study, including the specific processes or operations to be analyzed.
Gather and review all relevant documentation, such as process flow diagrams, piping and instrumentation diagrams, and standard operating procedures.

Execution

Divide the Process into Nodes: Break down the process into manageable sections or nodes. Each node typically represents a specific part of the process, such as a piece of equipment or a process step.
Identify Deviations: For each node, guidewords are applied to identify potential deviations from the intended design or operation. Common guidewords include:
- No: Complete absence of a process parameter (e.g., no flow).
- More: Quantitative increase (e.g., more pressure).
- Less: Quantitative decrease (e.g., less temperature).
- As well as: Presence of additional elements (e.g., contamination).
- Part of: Partial completion of an action (e.g., partial mixing).
- Reverse: Logical opposite of the intended action (e.g., reverse flow).
Analyze Causes and Consequences: Determine the possible causes of each deviation and analyze the potential consequences on safety, environment, and operations. This involves considering various factors such as equipment failure, human error, environmental conditions, or procedural issues that could lead to the deviation.
- Use of Experience and Knowledge: The team relies on their collective experience and knowledge of the process, equipment, and industry standards to hypothesize potential causes. This may include reviewing historical data, previous incidents, and near misses.
Recommend Actions: Develop recommendations for mitigating identified risks, such as changes to the process, additional controls, or procedural modifications.

Documentation and Follow-Up

Document all findings, including identified hazards, potential consequences, and recommended actions.
Assign responsibilities for implementing recommendations and establish timelines for completion.
Conduct follow-up reviews to ensure that recommended actions have been implemented effectively and that the process remains safe and operable.

Review and Update

Regularly review and update the HAZOP study to account for changes in processes, equipment, or regulations.
Ensure continuous improvement by incorporating lessons learned from past incidents or near misses.
Iterative Process: The process is iterative, with the team revisiting and refining their analysis as more information becomes available or as the understanding of the process deepens.

Node	Guideword	Parameter	Deviation	Cause	Consequence	Safeguards	Recommendations	Actions
Specific section or equipment being analyzed	Guideword applied (e.g. No, More, Less, Reverse, etc.)	Process parameter being examined (e.g. Flow, Temperature, Pressure, etc.)	How the parameter deviates from design intent when guideword is applied	Possible reasons for the deviation	Potential results if deviation occurs	Existing measures to prevent or mitigate the deviation	Suggested additional measures to control the risk	Specific tasks assigned to implement recommendations

Multi-Criteria Decision-Making to Drive Risk Control

To be honest, too often, we perform a risk assessment not to make decisions but to justify an already existing risk assessment. The risk assessment may help define a few additional action items and determine how rigorous to be about a few things. It actually didn’t make much of an impact on the already-decided path forward. This is some pretty bad risk management and decision-making.

For highly important decisions with high uncertainty or complexity, it is useful to consider the options/alternatives that exist and assess the benefits and risks of each before deciding on a path forward. Thoroughly identifying options/alternatives and assessing the benefits and risks of each can help the decision-making process and ultimately reduce risk.

An effective, highly structured decision-making process can help answer the question, ‘How can we compare the consequences of the various options before deciding?

The most challenging risk decisions are characterized by having several different, important things to consider in an environment where there are often multiple stakeholders and, often, multiple decision-makers.

In Multi-Criteria Decision-Making (MCDM), the primary objective is the structured consideration of the available alternatives (options) for achieving the objectives in order to make the most informed decision, leading to the best outcome.

In a Quality Risk Management context, the decision-making concerns making informed decisions in the face of uncertainty about risks related to the quality (and/or availability) of medicines.

Key Concepts of MCDM

Conflicting Criteria: MCDM deals with situations where criteria conflict. For example, when purchasing a car, one might need to balance cost, comfort, safety, and fuel economy, which often do not align perfectly.
Explicit Evaluation: Unlike intuitive decision-making, MCDM involves a structured approach to explicitly evaluate multiple criteria, which is crucial when the stakes are high, such as deciding whether to build additional manufacturing capacity for a product under development.
Types of Problems:

Multiple-Criteria Evaluation Problems: These involve a finite number of alternatives known at the beginning. The goal is to find the best alternative or a set of good alternatives based on their performance across multiple criteria.
Multiple-Criteria Design Problems: In these problems, alternatives are not explicitly known and must be found by solving a mathematical model. The number of alternatives can be very large, often exponentially.

Preference Information: The methods used in MCDM often require preference information from decision-makers (DMs) to differentiate between solutions. This can be done at various stages of the decision-making process, such as prior articulation of preferences, which transforms the problem into a single-criterion problem.

MCDM focuses on risk and uncertainty by explicitly weighing criteria and trade-offs between them. Multi-criteria decision-making (MCDM) differs from traditional decision-making methods in several key ways:

Explicit Consideration of Multiple Criteria: Traditional decision-making often focuses on a single criterion like cost or profit. MCDM explicitly considers multiple criteria simultaneously, which may be conflicting, such as cost, quality, safety, and environmental impact[1]. This allows for a more comprehensive evaluation of alternatives.
Structured Approach: MCDM provides a structured framework for evaluating alternatives against multiple criteria rather than relying solely on intuition or experience. It involves techniques like weighting criteria, scoring alternatives, and aggregating scores to rank or choose the best option.
Transparency and Consistency: MCDM methods aim to make decision-making more transparent, consistent, and less susceptible to individual biases. The criteria, weights, and evaluation process are explicitly defined, allowing for better justification and reproducibility of decisions.
Quantitative Analysis: Many MCDM methods employ quantitative techniques, such as mathematical models, optimization algorithms, and decision support systems. This enables a more rigorous and analytical approach compared to traditional qualitative methods.
Handling Complexity: MCDM is particularly useful for complex decision problems involving many alternatives, conflicting objectives, and multiple stakeholders. Traditional methods may struggle to handle such complexity effectively.
Stakeholder Involvement: Some MCDM methods, like the Analytic Hierarchy Process (AHP), facilitate the involvement of multiple stakeholders and the incorporation of their preferences and judgments. This can lead to more inclusive and accepted decisions.
Trade-off Analysis: MCDM techniques often involve analyzing trade-offs between criteria, helping decision-makers understand the implications of prioritizing certain criteria over others. This can lead to more informed and balanced decisions.

While traditional decision-making methods rely heavily on experience, intuition, and qualitative assessments, MCDM provides a more structured, analytical, and comprehensive approach, particularly in complex situations with conflicting criteria.

Multi-Criteria Decision-Making (MCDM) is typically performed following these steps:

Define the Decision Problem: Clearly state the problem or decision to be made, identify the stakeholders involved, and determine the desired outcome or objective.
Establish Criteria: Identify the relevant criteria that will be used to evaluate the alternatives. These criteria should be measurable, independent, and aligned with the objectives. Involve stakeholders in selecting and validating the criteria.
Generate Alternatives: Develop a comprehensive list of potential alternatives or options that could solve the problem. Use techniques like brainstorming, benchmarking, or scenario analysis to generate diverse alternatives.
Gather Performance Data: Assess how each alternative performs against each criterion. This may involve quantitative data, expert judgments, or qualitative assessments.
Assign Criteria Weights: By assigning weights, determine each criterion’s relative importance or priority. This can be done through methods like pairwise comparisons, swing weighting, or direct rating. Stakeholder input is crucial here.
Apply MCDM Method: Choose an appropriate MCDM technique based on the problem’s nature and the available data. Some popular methods include: Analytic Hierarchy Process (AHP); Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS); ELimination and Choice Expressing REality (ELECTRE); Preference Ranking Organization METHod for Enrichment of Evaluations (PROMETHEE); and, Multi-Attribute Utility Theory (MAUT).
Evaluate and Rank Alternatives: Apply the chosen MCDM method to evaluate and rank the alternatives based on their performance against the weighted criteria. This may involve mathematical models, software tools, or decision support systems.
Sensitivity Analysis: Perform sensitivity analysis to assess the robustness of the results and understand how changes in criteria weights or performance scores might affect the ranking or choice of alternatives.
Make the Decision: Based on the MCDM analysis, select the most preferred alternative or develop an action plan based on the ranking of alternatives. Involve stakeholders in the final decision-making process.
Monitor and Review: Implement the chosen alternative and monitor its performance. Review the decision periodically, and if necessary, repeat the MCDM process to adapt to changing circumstances or new information.

MCDM is an iterative process; stakeholder involvement, transparency, and clear communication are crucial. Additionally, the specific steps and techniques may vary depending on the problem’s complexity, the data’s availability, and the decision-maker’s preferences.

MCDM Technique	Description	Application	Key Features
Analytic Hierarchy Process (AHP)	A structured technique for organizing and analyzing complex decisions, using mathematics and psychology.	Widely used in business, government, and healthcare for prioritizing and decision-making.	Pairwise comparisons, consistency checks, and hierarchical structuring of criteria and alternatives.
Technique for Order Preference by Similarity to Ideal Solution (TOPSIS)	Based on the concept that the chosen alternative should have the shortest geometric distance from the positive ideal solution and the longest geometric distance from the negative ideal solution.	Frequently used in engineering, management, and human resource management for ranking and selection problems.	Compensatory aggregation, normalization of criteria, and calculation of geometric distances.
Elimination and Choice Expressing Reality (ELECTRE)	An outranking method that compares alternatives by considering both qualitative and quantitative criteria. It uses a pairwise comparison approach to eliminate less favorable alternatives.	Commonly used in project selection, resource allocation, and environmental management.	Use of concordance and discordance indices, handling of both qualitative and quantitative data, and ability to deal with incomplete rankings.
Preference Ranking Organization Method for Enrichment Evaluation (PROMETHEE)	An outranking method that uses preference functions to compare alternatives based on multiple criteria. It provides a complete ranking of alternatives.	Applied in various fields such as logistics, finance, and environmental management.	Preference functions, visual interactive modules (GAIA), and sensitivity analysis.
Multi-Attribute Utility Theory (MAUT)	Involves converting multiple criteria into a single utility function, which is then used to evaluate and rank alternatives. It takes into account the decision-maker’s risk preferences and uncertainties.	Used in complex decision-making scenarios involving risk and uncertainty, such as policy analysis and strategic planning.	Utility functions, probabilistic weights, and handling of uncertainty.

Popular MCDM Techniques