A critical step in ensuring the quality and safety of processes as part of verification is Design Review, which is sometimes expanded to Design Qualification.
Design Review is a systematic, documented examination of a proposed design to evaluate its adequacy and identify potential issues early in the development process. Here’s how to conduct an effective Design Review:
Plan Systematically: Schedule reviews at appropriate stages of development, ensuring they align with your project timeline.
Involve the Right People: Include representatives from all relevant functions and an independent reviewer not directly responsible for the design stage being evaluated.
Focus on Critical Aspects: Prioritize design elements that directly impact product quality and patient safety.
Document Thoroughly: Record all findings, including the design under review, participants, date, and any proposed actions.
Iterate as Needed: Conduct reviews iteratively as supplier design documents are published, allowing for early issue identification and correction.
Design Qualification: Verifying Suitability
Design Qualification (DQ) is the documented verification that the proposed design of facilities, equipment, or systems is suitable for its intended purpose. Here’s how to implement DQ effectively:
Develop User Requirements: Create a detailed User Requirements Specification (URS) outlining what the equipment or system is expected to do.
Create Functional Specifications: Translate user requirements into technical specifications that guide the design process.
Perform Risk Assessment: Identify potential risks associated with the design and develop mitigation strategies.
Review Design Specifications: Ensure the design meets all specified requirements, including GMP and regulatory standards.
Document and Approve: Formally document the DQ process and obtain approval from key stakeholders, including quality assurance personnel.
Integrating Design Review and DQ
To maximize the effectiveness of these processes:
Use a Risk-Based Approach: Prioritize efforts based on the level of risk to product quality and patient safety.
Leverage Subject Matter Experts: Involve SMEs from the start to contribute their expertise throughout the process.
Implement Change Management: Establish a robust system to manage design changes effectively and avoid late-stage issues.
Ensure Quality Oversight: Have Quality Assurance provide oversight to maintain compliance with current regulations and GMP requirements.
Document Comprehensively: Maintain thorough records of all reviews, qualifications, and decisions made during the process.
Implementing a systematic approach to Design Review and Design Qualification not only helps meet regulatory expectations but also contributes to operational efficiency and product excellence. As the pharmaceutical landscape evolves, staying committed to these foundational practices will remain crucial for success in this highly regulated industry.
User requirements are typically divided into several categories to ensure comprehensive coverage of all aspects of product development, manufacturing, and quality control and to help guide the risk-based approach to verification.
Quality requirements focus on ensuring that the product meets all necessary quality standards and regulatory compliance. This category includes:
Good Manufacturing Practices (GMP) compliance, including around cleaning, cross-contamination, etc to ensure compliance with various regulations such as FDA guidelines, EU GMP, and ICH standards.
Documentation and record-keeping standards
Contamination control strategies are a key part of quality requirements, as they are essential for maintaining product quality and patient safety.
Data integrity requirements fall under this category, as they are crucial for ensuring the quality and reliability of data.
Not everyone advocates for this breakdown but I am a huge proponent as it divides the product specific requirements for the more standard must’s of meeting the cGMPs that are not product specific. This really helps when you are a multi-product facility and it helps define what is in the PQ versus what is in the PPQ.
Safety User Requirements
Safety requirements address the safety of personnel, patients, and the environment. They encompass:
Occupational health and safety measures
Environmental protection protocols
Patient safety considerations in product design
General User Requirements
General requirements cover broader aspects of the manufacturing system and facility. These may include:
Facility design and layout
Equipment specifications
Utility requirements (e.g., power, water, HVAC)
Maintenance procedures
By categorizing user requirements in this way, pharmaceutical companies can ensure a comprehensive approach to product development and manufacturing, addressing all critical aspects from product quality to regulatory compliance and safety. This will help drive appropriate verification.
“The specification for equipment, facilities, utilities or systems should be defined in a URS and/or a functional specification. The essential elements of quality need to be built in at this stage and any GMP risks mitigated to an acceptable level. The URS should be a point of reference throughout the validation life cycle.” – Annex 15, Section 3.2, Eudralex Volume 4
User Requirement Specifications serve as a cornerstone of quality in pharmaceutical manufacturing. They are not merely bureaucratic documents but vital tools that ensure the safety, efficacy, and quality of pharmaceutical products.
Defining the Essentials
A well-crafted URS outlines the critical requirements for facilities, equipment, utilities, systems and processes in a regulated environment. It captures the fundamental aspects and scope of users’ needs, ensuring that all stakeholders have a clear understanding of what is expected from the final product or system.
The phrase “essential elements of quality need to be built in at this stage” emphasizes the proactive approach to quality assurance. By incorporating quality considerations from the outset, manufacturers can:
Reduce the need for costly corrections later in the process
Ensure compliance with Good Manufacturing Practice (GMP) standards
Mitigating GMP Risks
Risk management is a crucial aspect of pharmaceutical manufacturing. The URS plays a vital role in identifying and addressing potential GMP risks early in the development process. By doing so, manufacturers can:
Ensure that the final product meets regulatory requirements
The URS as a Living Document
One of the key points in the regulations is that the URS should be “a point of reference throughout the validation life cycle.” This underscores the dynamic nature of the URS and its ongoing importance.
Continuous Reference
Throughout the development, implementation, and operation of a system or equipment, the URS serves as:
A benchmark for assessing progress
A guide for making decisions
A tool for resolving disputes or clarifying requirements
Adapting to Change
As projects evolve, the URS may need to be updated to reflect new insights, technological advancements, or changing regulatory requirements. This flexibility ensures that the final product remains aligned with user needs and regulatory expectations.
Practical Implications
Involve multidisciplinary teams in creating the URS, including representatives from quality assurance, engineering, production, and regulatory affairs.
Conduct thorough risk assessments to identify potential GMP risks and incorporate mitigation strategies into the URS.
Ensure clear, objectively stated requirements that are verifiable during testing and commissioning.
Align the URS with company objectives and strategies to ensure long-term relevance and support.
Implement robust version control and change management processes for the URS throughout the validation lifecycle.
Executing the Control Space from the Design Space
The User Requirements Specification (URS) is a mechanism for executing the control space, from the design space as outlined in ICH Q8. To understand that, let’s discuss the path from a Quality Target Product Profile (QTPP) to Critical Quality Attributes (CQAs) to Critical Process Parameters (CPPs) with Proven Acceptable Ranges (PARs), which is a crucial journey in pharmaceutical development using Quality by Design (QbD) principles. This systematic approach ensures that the final product meets the desired quality standards and user needs.
It is important to remember that this is usually a set of user requirements specifications, respecting the system boundaries.
From QTPP to CQAs
The journey begins with defining the Quality Target Product Profile (QTPP). The QTPP is a comprehensive summary of the quality characteristics that a drug product should possess to ensure its safety, efficacy, and overall quality. It serves as the foundation for product development and includes considerations such as:
Dosage strength
Delivery system
Dosage form
Container system
Purity
Stability
Sterility
Once the QTPP is established, the next step is to identify the Critical Quality Attributes (CQAs). CQAs are physical, chemical, biological, or microbiological properties that should be within appropriate limits to ensure the desired product quality. These attributes are derived from the QTPP and are critical to the safety and efficacy of the product.
From CQAs to CPPs
With the CQAs identified, the focus shifts to determining the Critical Process Parameters (CPPs). CPPs are process variables that have a direct impact on the CQAs. These parameters must be monitored and controlled to ensure that the product consistently meets the desired quality standards. Examples of CPPs include:
Temperature
pH
Cooling rate
Rotation speed
The relationship between CQAs and CPPs is established through risk assessment, experimentation, and data analysis. This step often involves Design of Experiments (DoE) to understand how changes in CPPs affect the CQAs. This is Process Characterization.
Establishing PARs
For each CPP, a Proven Acceptable Range (PAR) is determined. The PAR represents the operating range within which the CPP can vary while still ensuring that the CQAs meet the required specifications. PARs are established through rigorous testing and validation processes, often utilizing statistical tools and models.
Build the Requirements for the CPPs
The CPPs with PARs are process parameters that can affect critical quality attributes of the product and must be controlled within predetermined ranges. These are translated into user requirements. Many will specifically label these as Product User Requirements (PUR) to denote they are linked to the overall product capability. This helps to guide risk assessments and develop an overall verification approach.
Most of Us End Up on the Less than Happy Path
This approach is the happy path that aligns nicely with the FDA’s Process Validation Model.
This can quickly break down in the real world. Most of us go into CDMOs with already qualified equipment. We have platforms on which we’ve qualified our equipment, too. We don’t know the CPPs until just before PPQ.
This makes the user requirements even more important as living documents. Yes, we’ve qualified our equipment for these large ranges. Now that we have the CPPs, we update the user requirements for the Product User Requirements, perform an overall assessment of the gaps, and, with a risk-based approach, do additional verification activations either before or as part of Process Performance Qualification (PPQ).
ICH Q9(R1) emphasizes that knowledge is fundamental to effective risk management. The guideline states that “QRM is part of building knowledge and understanding risk scenarios, so that appropriate risk control can be decided upon for use during the commercial manufacturing phase.”
We need to recognize the inverse relationship between knowledge and uncertainty in risk assessment. ICH Q9(R1) notes that uncertainty may be reduced “via effective knowledge management, which enables accumulated and new information (both internal and external) to be used to support risk-based decisions throughout the product lifecycle”
In order to gauge the confidence in risk assessment we need to gauge our knowledge strength.
The Spectrum of Knowledge Strength
Knowledge strength can be categorized into three levels: weak, medium, and strong. Each level is determined by specific criteria that assess the reliability, consensus, and depth of understanding surrounding a particular subject.
Indicators of Weak Knowledge
Knowledge is considered weak if it exhibits one or more of the following characteristics:
Oversimplified Assumptions: The foundations of the knowledge rely on strong simplifications that may not accurately represent reality.
Lack of Reliable Data: There is little to no data available, or the existing information is highly unreliable or irrelevant.
Expert Disagreement: There is significant disagreement among experts in the field.
Poor Understanding of Phenomena: The underlying phenomena are poorly understood, and available models are either non-existent or known to provide inaccurate predictions.
Unexamined Knowledge: The knowledge has not been thoroughly scrutinized, potentially overlooking critical “unknown knowns.”
Hallmarks of Strong Knowledge
On the other hand, knowledge is deemed strong when it meets all of the following criteria (where relevant):
Reasonable Assumptions: The assumptions made are considered very reasonable and well-grounded.
Abundant Reliable Data: Large amounts of reliable and relevant data or information are available.
Expert Consensus: There is broad agreement among experts in the field.
Well-Understood Phenomena: The phenomena involved are well understood, and the models used provide predictions with the required accuracy.
Thoroughly Examined: The knowledge has been rigorously examined and tested.
The Middle Ground: Medium Strength Knowledge
Cases that fall between weak and strong are classified as medium strength knowledge. This category can be flexible, allowing for a broader range of scenarios to be considered strong. For example, knowledge could be classified as strong if at least one (or more) of the strong criteria are met while none of the weak criteria are present.
Strong vs Weak Knowledge
A Simplified Approach
For practical applications, a simplified version of this framework can be used:
Strong: All criteria for strong knowledge are met.
Medium: One or two criteria for strong knowledge are not met.
Weak: Three or more criteria for strong knowledge are not met.
Implications for Decision-Making
Understanding the strength of our knowledge is crucial for effective decision-making. Strong knowledge provides a solid foundation for confident choices, while weak knowledge signals the need for caution and further investigation.
When faced with weak knowledge:
Seek additional information or expert opinions
Consider multiple scenarios and potential outcomes
Implement risk mitigation strategies
When working with strong knowledge:
Make decisions with greater confidence
Focus on implementation and optimization
Monitor outcomes to validate and refine understanding
Strong knowledge typically corresponds to lower levels of uncertainty:
Level 1 Uncertainty: This aligns closely with strong knowledge, where outcomes can be estimated with reasonable accuracy within a single system model. Strong knowledge is characterized by reasonable assumptions, abundant reliable data, and well-understood phenomena, which enable accurate predictions.
Level 2 Uncertainty: While displaying alternative futures, this level still operates within a single system where probability estimates can be applied confidently. Strong knowledge often allows for this level of certainty, as it involves broad expert agreement and thoroughly examined information.
Medium Knowledge and Moderate Uncertainty (Level 3)
Medium strength knowledge often corresponds to Level 3 uncertainty:
Level 3 Uncertainty: This level involves “a multiplicity of plausible futures” with multiple interacting systems, but still within a known range of outcomes. Medium knowledge strength might involve some gaps or disagreements but still provides a foundation for identifying potential outcomes.
Weak Knowledge and Deep Uncertainty (Level 4)
Weak knowledge aligns most closely with the deepest level of uncertainty:
Level 4 Uncertainty: This level leads to an “unknown future” where we don’t understand the system and are aware of crucial unknowns. Weak knowledge, characterized by oversimplified assumptions, lack of reliable data, and poor understanding of phenomena, often results in this level of deep uncertainty.
Implications for Decision-Making
When knowledge is strong and uncertainty is low (Levels 1-2), decision-makers can rely more confidently on predictions and probability estimates.
As knowledge strength decreases and uncertainty increases (Levels 3-4), decision-makers must adopt more flexible and adaptive approaches to account for a wider range of possible futures.
The principle that “uncertainty should always be considered at the deepest proposed level” unless proven otherwise aligns with the cautious approach of assessing knowledge strength. This ensures that potential weaknesses in knowledge are not overlooked.
Conclusion
By systematically evaluating the strength of our knowledge using this framework, we can make more informed decisions, identify areas that require further investigation, and better understand the limitations of our current understanding. Remember, the goal is not always to achieve perfect knowledge but to recognize the level of certainty we have and act accordingly.
Twenty years on, risk management in the pharmaceutical world continues to be challenging. Ensure that risk assessments are systematic, structured, and based on scientific knowledge. A large part of the ICH Q9(R1) revision was written to address continued struggles with subjectivity, formality, and decision-making. And quite frankly, it’s clear to me that we, as an industry, are still working to absorb those messages these last two years.
A big challenge is that we struggle to measure the effectiveness of our risk assessments. Quite frankly, this is a great place for a rubric.
Luckily, we have a good tool out there to adopt: the Risk Analysis Quality Test (RAQT1.0), developed by the Society for Risk Analysis (SRA). This comprehensive framework is designed to evaluate and improve the quality of risk assessments. We can apply this tool to meet the requirements of the International Conference on Harmonisation (ICH) Q9, which outlines quality risk management principles for the pharmaceutical industry. From that, we can drive continued improvement in our risk management activities.
Components of RAQT1.0
The Risk Analysis Quality Test consists of 76 questions organized into 15 categories:
Framing the Analysis and Its Interface with Decision Making
Capturing the Risk Generating Process (RGP)
Communication
Stakeholder Involvement
Assumptions and Scope Boundary Issues
Proactive Creation of Alternative Courses of Action
Basis of Knowledge
Data Limitations
Analysis Limitations
Uncertainty
Consideration of Alternative Analysis Approaches
Robustness and Resilience of Action Strategies
Model and Analysis Validation and Documentation
Reporting
Budget and Schedule Adequacy
Application to ICH Q9 Requirements
ICH Q9 emphasizes the importance of a systematic and structured risk assessment process. The RAQT can be used to ensure that risk assessments are thorough and meet quality standards. For example, Category G (Basis of Knowledge) and Category H (Data Limitations) help in evaluating the scientific basis and data quality of the risk assessment, aligning with ICH Q9’s requirement for using available knowledge and data.
The RAQT’s Category B (Capturing the Risk Generating Process) and Category C (Communication) can help in identifying and communicating risks effectively. This aligns with ICH Q9’s requirement to identify potential risks based on scientific knowledge and understanding of the process.
Categories such as Category I (Analysis Limitations) and Category J (Uncertainty) in the RAQT help in analyzing the risks and addressing uncertainties, which is a key aspect of ICH Q9. These categories ensure that the analysis is robust and considers all relevant factors.
The RAQT’s Category A (Framing the Analysis and Its Interface with Decision Making) and Category F (Proactive Creation of Alternative Courses of Action) are crucial for evaluating risks and developing mitigation strategies. This aligns with ICH Q9’s requirement to evaluate risks and determine the need for risk reduction.
Categories like Category L (Robustness and Resilience of Action Strategies) and Category M (Model and Analysis Validation and Documentation) in the RAQT help in ensuring that the risk control measures are robust and well-documented. This is consistent with ICH Q9’s emphasis on implementing and reviewing controls.
Category D (Stakeholder Involvement) of the RAQT ensures that stakeholders are engaged in the risk management process, which is a requirement under ICH Q9 for effective communication and collaboration.
The RAQT can be applied both retrospectively and prospectively, allowing for the evaluation of past risk assessments and the planning of future ones. This aligns with ICH Q9’s requirement for periodic review and continuous improvement of the risk management process.
Creating a Rubric
To make this actionable we need a tool, a rubric, to allow folks to evaluate what goods look like. I would insert this tool into the quality oversite of risk management.
Category A: Framing the Analysis and Its Interface With Decision Making
Criteria
Excellent (4)
Good (3)
Fair (2)
Poor (1)
Problem Definition
Clearly and comprehensively defines the problem, including all relevant aspects and stakeholders
Adequately defines the problem with most relevant aspects considered
Partially defines the problem with some key aspects missing
Poorly defines the problem or misses critical aspects
Analytical Approach
Selects and justifies an optimal analytical approach, demonstrating deep understanding of methodologies
Chooses an appropriate analytical approach with reasonable justification
Selects a somewhat relevant approach with limited justification
Chooses an inappropriate approach or provides no justification
Data Collection and Management
Thoroughly identifies all necessary data sources and outlines a comprehensive data management plan
Identifies most relevant data sources and provides a adequate data management plan
Identifies some relevant data sources and offers a basic data management plan
Fails to identify key data sources or lacks a coherent data management plan
Stakeholder Identification
Comprehensively identifies all relevant stakeholders and their interests
Identifies most key stakeholders and their primary interests
Identifies some stakeholders but misses important ones or their interests
Fails to identify major stakeholders or their interests
Decision-Making Context
Provides a thorough analysis of the decision-making context, including constraints and opportunities
Adequately describes the decision-making context with most key factors considered
Partially describes the decision-making context, missing some important factors
Poorly describes or misunderstands the decision-making context
Alignment with Organizational Goals
Demonstrates perfect alignment between the analysis and broader organizational objectives
Shows good alignment with organizational goals, with minor gaps
Partially aligns with organizational goals, with significant gaps
Fails to align with or contradicts organizational goals
Communication Strategy
Develops a comprehensive strategy for communicating results to all relevant decision-makers
Outlines a good communication strategy covering most key decision-makers
Provides a basic communication plan with some gaps
Lacks a clear strategy for communicating results to decision-makers
This rubric provides a framework for assessing the quality of work in framing an analysis and its interface with decision-making. It covers key aspects such as problem definition, analytical approach, data management, stakeholder consideration, decision-making context, alignment with organizational goals, and communication strategy. Each criterion is evaluated on a scale from 1 (Poor) to 4 (Excellent), allowing for nuanced assessment of performance in each area.
To use this rubric effectively:
Adjust the criteria and descriptions as needed to fit your specific context or requirements.
Ensure that the expectations for each level (Excellent, Good, Fair, Poor) are clear and distinguishable.
My next steps will be to add specific examples or indicators for each level to provide more guidance to both assessors and those being assessed.
I also may, depending on internal needs, want to assign different weights to each criterion based on their relative importance in your specific context. In this case I think each ends up being pretty similar.
I would then go and add the other sections. For example, here is category B with some possible weighting.
Category B: Capturing the Risk Generating Process (RGP)
Component
Weight Factor
Excellent
Satisfactory
Needs Improvement
Poor
B1. Comprehensiveness
4
The analysis includes: i) A structured taxonomy of hazards/events demonstrating comprehensiveness ii) Each scenario spelled out with causes and types of change iii) Explicit addressing of potential “Black Swan” events iv) Clear description of implications of such events for risk management
The analysis includes 3 out of 4 elements from the Excellent criteria, with minor gaps that do not significantly impact understanding
The analysis includes only 2 out of 4 elements from the Excellent criteria, or has significant gaps in comprehensiveness
The analysis includes 1 or fewer elements from the Excellent criteria, severely lacking in comprehensiveness
B2. Basic Structure of RGP
2
Clearly identifies and accounts for the basic structure of the RGP (e.g. linear, chaotic, complex adaptive) AND Uses appropriate mathematical structures (e.g. linear, quadratic, exponential) that match the RGP structure
Identifies the basic structure of the RGP BUT does not fully align mathematical structures with the RGP
Attempts to identify the RGP structure but does so incorrectly or incompletely OR Uses mathematical structures that do not align with the RGP
Does not identify or account for the basic structure of the RGP
B3. Complexity of RGP
3
Lists all important causal and associative links in the RGP AND Demonstrates how each link is accounted for in the analysis
Lists most important causal and associative links in the RGP AND Demonstrates how most links are accounted for in the analysis
Lists some causal and associative links but misses key elements OR Does not adequately demonstrate how links are accounted for in the analysis
Does not list causal and associative links or account for them in the analysis
B4. Early Warning Detection
3
Includes a clear process for detecting early warnings of potential surprising risk aspects, beyond just concrete events
Includes a process for detecting early warnings, but it may be limited in scope or not fully developed
Mentions the need for early warning detection but does not provide a clear process
Does not address early warning detection
B5. System Changes
2
Fully considers the possibility of system changes AND Establishes adequate mechanisms to detect those changes
Considers the possibility of system changes BUT mechanisms to detect changes are not fully developed
Mentions the possibility of system changes but does not adequately consider or establish detection mechanisms
Does not consider or address the possibility of system changes
I definitely need to go back and add more around structure requirements. The SRA RAQT tool needs some more interpretation here.
Category C: Risk Communication
Component
Weight Factor
Excellent
Satisfactory
Needs Improvement
Poor
C1. Integration of Communication into Risk Analysis
3
Communication is fully integrated into the risk analysis following established norms). All aspects of the methodology are clearly addressed including context establishment, risk assessment (identification, analysis, evaluation), and risk treatment. There is clear evidence of pre-assessment, management, appraisal, characterization and evaluation. Knowledge about the risk is thoroughly categorized.
Communication is integrated into the risk analysis following most aspects of established norms. Most key elements of methodologies like ISO 31000 or IRGC are addressed, but some minor aspects may be missing or unclear. Knowledge about the risk is categorized, but may lack some detail.
Communication is partially integrated into the risk analysis, but significant aspects of established norms are missing. Only some elements of methodologies like ISO 31000 or IRGC are addressed. Knowledge categorization about the risk is incomplete or unclear.
There is little to no evidence of communication being integrated into the risk analysis following established norms. Methodologies like ISO 31000 or IRGC are not followed. Knowledge about the risk is not categorized.
C2. Adequacy of Risk Communication
3
All considerations for effective risk communication have been applied to ensure adequacy between analysts and decision makers, analysts and other stakeholders, and decision makers and stakeholders. There is clear evidence that all parties agree the communication is adequate.
Most considerations for effective risk communication have been applied. Communication appears adequate between most parties, but there may be minor gaps or areas where agreement on adequacy is not explicitly stated.
Some considerations for effective risk communication have been applied, but there are significant gaps. Communication adequacy is questionable between one or more sets of parties. There is limited evidence of agreement on communication adequacy.
Few to no considerations for effective risk communication have been applied. There is no evidence of adequate communication between analysts, decision makers, and stakeholders. There is no indication of agreement on communication adequacy.
Category D: Stakeholder Involvement
Criteria
Weight
Excellent (4)
Satisfactory (3)
Needs Improvement (2)
Poor (1)
Stakeholder Identification
4
All relevant stakeholders are systematically and comprehensively identified
Most relevant stakeholders are identified, with minor omissions
Some relevant stakeholders are identified, but significant groups are missed
Few or no relevant stakeholders are identified
Stakeholder Consultation
3
All identified stakeholders are thoroughly consulted, with their perceptions and concerns fully considered
Most identified stakeholders are consulted, with their main concerns considered
Some stakeholders are consulted, but consultation is limited in scope or depth
Few or no stakeholders are consulted
Stakeholder Engagement
3
Stakeholders are actively engaged throughout the entire risk management process, including problem framing, decision-making, and implementation
Stakeholders are engaged in most key stages of the risk management process
Stakeholders are engaged in some aspects of the risk management process, but engagement is inconsistent
Stakeholders are minimally engaged or not engaged at all in the risk management process
Effectiveness of Involvement
2
All stakeholders would agree that they were effectively consulted and engaged
Most stakeholders would agree that they were adequately consulted and engaged
Some stakeholders may feel their involvement was insufficient or ineffective
Most stakeholders would likely feel their involvement was inadequate or ineffective
Category E: Assumptions and Scope Boundary Issues
Criterion
Weight
Excellent (4)
Satisfactory (3)
Needs Improvement (2)
Poor (1)
E1. Important assumptions and implications listed
4
All important assumptions and their implications for risk management are systematically listed in clear language understandable to decision makers. Comprehensive and well-organized.
Most important assumptions and implications are listed in language generally clear to decision makers. Some minor omissions or lack of clarity.
Some important assumptions and implications are listed, but significant gaps exist. Language is not always clear to decision makers.
Few or no important assumptions and implications are listed. Language is unclear or incomprehensible to decision makers.
E2. Risks of assumption deviations evaluated
3
Risks of all significant assumptions deviating from the actual Risk Generating Process are thoroughly evaluated. Consequences and implications are clearly communicated to decision makers.
Most risks of significant assumption deviations are evaluated. Consequences and implications are generally communicated to decision makers, with minor gaps.
Some risks of assumption deviations are evaluated, but significant gaps exist. Communication to decision makers is incomplete or unclear.
Few or no risks of assumption deviations are evaluated. Little to no communication of consequences and implications to decision makers.
E3. Scope boundary issues and implications listed
3
All important scope boundary issues and their implications for risk management are systematically listed in clear language understandable to decision makers. Comprehensive and well-organized.
Most important scope boundary issues and implications are listed in language generally clear to decision makers. Some minor omissions or lack of clarity.
Some important scope boundary issues and implications are listed, but significant gaps exist. Language is not always clear to decision makers.
Few or no important scope boundary issues and implications are listed. Language is unclear or incomprehensible to decision makers.
Category F: Proactive Creation of Alternative Courses of Action
Criteria
Weight
Excellent (4)
Satisfactory (3)
Needs Improvement (2)
Poor (1)
Systematic generation of alternatives
4
A comprehensive and structured process is used to systematically generate a wide range of alternative courses of action, going well beyond initially considered options
A deliberate process is used to generate multiple alternative courses of action beyond those initially considered
Some effort is made to generate alternatives, but the process is not systematic or comprehensive
Little to no effort is made to generate alternatives beyond those initially considered
Goal-focused creation
3
All generated alternatives are clearly aligned with and directly address the stated goals of the analysis
Most generated alternatives align with the stated goals of the analysis
Some generated alternatives align with the goals, but others seem tangential or unrelated
Generated alternatives (if any) do not align with or address the stated goals
Consideration of robust/resilient options
3
Multiple robust and resilient alternatives are developed to address various uncertainty scenarios
At least one robust or resilient alternative is developed to address uncertainty
Robustness and resilience are considered, but not fully incorporated into alternatives
Robustness and resilience are not considered in alternative generation
Examination of unintended consequences
2
Thorough examination of potential unintended consequences for each alternative, including action-reaction spirals
Some examination of potential unintended consequences for most alternatives
Limited examination of unintended consequences for some alternatives
No consideration of potential unintended consequences
Documentation of alternative creation process
1
The process of alternative generation is fully documented, including rationale for each alternative
The process of alternative generation is mostly documented
The process of alternative generation is partially documented
The process of alternative generation is not documented
Category G: Basis of Knowledge
Criterion
Weight
Excellent (4)
Satisfactory (3)
Needs Improvement (2)
Poor (1)
G1. Characterization of knowledge basis
4
All inputs are clearly characterized (empirical, expert elicitation, testing, modeling, etc.). Distinctions between broadly accepted and novel analyses are explicitly stated.
Most inputs are characterized, with some minor omissions. Distinctions between accepted and novel analyses are mostly clear.
Some inputs are characterized, but significant gaps exist. Limited distinction between accepted and novel analyses.
Little to no characterization of knowledge basis. No distinction between accepted and novel analyses.
G2. Strength of knowledge adequacy
3
Strength of knowledge is thoroughly characterized in terms of its adequacy to support risk management decisions. Limitations are clearly articulated.
Strength of knowledge is mostly characterized, with some minor gaps in relating to decision support adequacy.
Limited characterization of knowledge strength. Unclear how it relates to decision support adequacy.
No characterization of knowledge strength or its adequacy for decision support.
G3. Communication of knowledge limitations
4
All knowledge limitations and their implications for risk management are clearly communicated to decision makers in understandable language.
Most knowledge limitations and implications are communicated, with minor clarity issues.
Some knowledge limitations are communicated, but significant gaps exist in clarity or completeness.
Knowledge limitations are not communicated or are presented in a way decision makers cannot understand.
G4. Consideration of surprises and unforeseen events
3
Thorough consideration of potential surprises and unforeseen events (Black Swans). Their importance is clearly articulated.
Consideration of surprises and unforeseen events is present, with some minor gaps in articulating their importance.
Limited consideration of surprises and unforeseen events. Their importance is not clearly articulated.
No consideration of surprises or unforeseen events.
G5. Conflicting expert opinions
2
All conflicting expert opinions are systematically considered and reported to decision makers as a source of uncertainty.
Most conflicting expert opinions are considered and reported, with minor omissions.
Some conflicting expert opinions are considered, but significant gaps exist in reporting or consideration.
Conflicting expert opinions are not considered or reported.
G6. Consideration of unconsidered knowledge
2
Explicit measures are implemented to check for knowledge outside the analysis group (e.g., independent review).
Some measures are in place to check for outside knowledge, but they may not be comprehensive.
Limited consideration of knowledge outside the analysis group. No formal measures in place.
No consideration of knowledge outside the analysis group.
G7. Consideration of disregarded low-probability events
1
Explicit measures are implemented to check for events disregarded due to low probabilities based on critical assumptions.
Some consideration of low-probability events, but measures may not be comprehensive.
Limited consideration of low-probability events. No formal measures in place.
No consideration of events disregarded due to low probabilities.
This rubric, once done, is a tool to guide assessment and provide feedback. It should be flexible enough to accommodate unique aspects of individual work while maintaining consistent standards across evaluations. I’d embed it in the quality approval step.
Investigations of a Dog 