Assessing the Strength of Knowledge: A Framework for Decision-Making

ICH Q9(R1) emphasizes that knowledge is fundamental to effective risk management. The guideline states that “QRM is part of building knowledge and understanding risk scenarios, so that appropriate risk control can be decided upon for use during the commercial manufacturing phase.”

We need to recognize the inverse relationship between knowledge and uncertainty in risk assessment. ICH Q9(R1) notes that uncertainty may be reduced “via effective knowledge management, which enables accumulated and new information (both internal and external) to be used to support risk-based decisions throughout the product lifecycle”

In order to gauge the confidence in risk assessment we need to gauge our knowledge strength.

The Spectrum of Knowledge Strength

Knowledge strength can be categorized into three levels: weak, medium, and strong. Each level is determined by specific criteria that assess the reliability, consensus, and depth of understanding surrounding a particular subject.

Indicators of Weak Knowledge

Knowledge is considered weak if it exhibits one or more of the following characteristics:

Oversimplified Assumptions: The foundations of the knowledge rely on strong simplifications that may not accurately represent reality.
Lack of Reliable Data: There is little to no data available, or the existing information is highly unreliable or irrelevant.
Expert Disagreement: There is significant disagreement among experts in the field.
Poor Understanding of Phenomena: The underlying phenomena are poorly understood, and available models are either non-existent or known to provide inaccurate predictions.
Unexamined Knowledge: The knowledge has not been thoroughly scrutinized, potentially overlooking critical “unknown knowns.”

Hallmarks of Strong Knowledge

On the other hand, knowledge is deemed strong when it meets all of the following criteria (where relevant):

Reasonable Assumptions: The assumptions made are considered very reasonable and well-grounded.
Abundant Reliable Data: Large amounts of reliable and relevant data or information are available.
Expert Consensus: There is broad agreement among experts in the field.
Well-Understood Phenomena: The phenomena involved are well understood, and the models used provide predictions with the required accuracy.
Thoroughly Examined: The knowledge has been rigorously examined and tested.

The Middle Ground: Medium Strength Knowledge

Cases that fall between weak and strong are classified as medium strength knowledge. This category can be flexible, allowing for a broader range of scenarios to be considered strong. For example, knowledge could be classified as strong if at least one (or more) of the strong criteria are met while none of the weak criteria are present.

A Simplified Approach

For practical applications, a simplified version of this framework can be used:

Strong: All criteria for strong knowledge are met.
Medium: One or two criteria for strong knowledge are not met.
Weak: Three or more criteria for strong knowledge are not met.

Implications for Decision-Making

Understanding the strength of our knowledge is crucial for effective decision-making. Strong knowledge provides a solid foundation for confident choices, while weak knowledge signals the need for caution and further investigation.

When faced with weak knowledge:

Seek additional information or expert opinions
Consider multiple scenarios and potential outcomes
Implement risk mitigation strategies

When working with strong knowledge:

Make decisions with greater confidence
Focus on implementation and optimization
Monitor outcomes to validate and refine understanding

Knowledge Strength and Uncertainty

The concept of knowledge strength aligns closely with the levels of uncertainty.

Strong Knowledge and Low Uncertainty (Levels 1-2)

Strong knowledge typically corresponds to lower levels of uncertainty:

Level 1 Uncertainty: This aligns closely with strong knowledge, where outcomes can be estimated with reasonable accuracy within a single system model. Strong knowledge is characterized by reasonable assumptions, abundant reliable data, and well-understood phenomena, which enable accurate predictions.
Level 2 Uncertainty: While displaying alternative futures, this level still operates within a single system where probability estimates can be applied confidently. Strong knowledge often allows for this level of certainty, as it involves broad expert agreement and thoroughly examined information.

Medium Knowledge and Moderate Uncertainty (Level 3)

Medium strength knowledge often corresponds to Level 3 uncertainty:

Level 3 Uncertainty: This level involves “a multiplicity of plausible futures” with multiple interacting systems, but still within a known range of outcomes. Medium knowledge strength might involve some gaps or disagreements but still provides a foundation for identifying potential outcomes.

Weak Knowledge and Deep Uncertainty (Level 4)

Weak knowledge aligns most closely with the deepest level of uncertainty:

Level 4 Uncertainty: This level leads to an “unknown future” where we don’t understand the system and are aware of crucial unknowns. Weak knowledge, characterized by oversimplified assumptions, lack of reliable data, and poor understanding of phenomena, often results in this level of deep uncertainty.

Implications for Decision-Making

When knowledge is strong and uncertainty is low (Levels 1-2), decision-makers can rely more confidently on predictions and probability estimates.
As knowledge strength decreases and uncertainty increases (Levels 3-4), decision-makers must adopt more flexible and adaptive approaches to account for a wider range of possible futures.
The principle that “uncertainty should always be considered at the deepest proposed level” unless proven otherwise aligns with the cautious approach of assessing knowledge strength. This ensures that potential weaknesses in knowledge are not overlooked.

Conclusion

By systematically evaluating the strength of our knowledge using this framework, we can make more informed decisions, identify areas that require further investigation, and better understand the limitations of our current understanding. Remember, the goal is not always to achieve perfect knowledge but to recognize the level of certainty we have and act accordingly.

Prioritization: MoSCoW, Binary and Pairwise

Prioritization tools are essential for effective decision-making. They help teams decide where to focus their efforts, ensuring that the most critical tasks are completed first.

MoSCoW Prioritization

The MoSCoW method is a widely used prioritization technique in project management, particularly within agile frameworks. It categorizes tasks or requirements into four distinct categories:

Must Have: Essential requirements that are critical for the project’s success. Without these, the project is considered a failure.
Should Have: Important but not critical requirements. These can be deferred if necessary but should be included if possible.
Could Have: Desirable but not necessary requirements. These are nice-to-haves that can be included if time and resources permit.
Won’t Have: Requirements agreed to be excluded from the current project scope. These might be considered for future phases.

Advantages:

Clarity and Focus: Clearly distinguish between essential and non-essential requirements, helping teams focus on what truly matters.
Stakeholder Alignment: Facilitates discussions and alignment among stakeholders regarding priorities.
Flexibility: Can be adapted to various project types and industries.

Disadvantages:

Ambiguity: May not provide clear guidance on prioritizing within each category.
Subjectivity: Decisions can be influenced by stakeholder biases or political considerations.
Resource Allocation: Requires careful allocation of resources to ensure that “Must Have” items are prioritized appropriately.

Binary Prioritization

Binary prioritization, often implemented using a binary search tree, is a method for systematically comparing and ranking requirements. Each requirement is compared against others, creating a hierarchical list of priorities.

Process:

Root Node: Start with one requirement as the root node.
Comparison: Compare each succeeding requirement to the root node, establishing child nodes based on priority.
Hierarchy: Continue creating a long list of prioritized requirements, forming a binary tree structure.

Advantages:

Systematic Approach: Provides a clear, structured way to compare and rank requirements.
Granularity: Offers detailed prioritization, ensuring that each requirement is evaluated against others.
Objectivity: Reduces subjectivity by using a consistent comparison method.

Disadvantages:

Complexity: Can be complex and time-consuming, especially for large projects with many requirements.
Resource Intensive: Requires significant effort to compare each requirement systematically.
Scalability: It may become unwieldy with many requirements, making it difficult to manage.

Pairwise Comparison

Pairwise or paired comparison is a method for prioritizing and ranking multiple options by comparing them in pairs. This technique is particularly useful when quantitative, objective data is not available, and decisions need to be made based on subjective criteria.

How Pairwise Comparison Works

Define Criteria: Establish clear criteria for evaluation, such as cost, strategic importance, urgency, resource allocation, or alignment with objectives.
Create a Matrix: List all the items to be compared along its rows and columns. Each cell in the matrix represents a comparison between two items.
Make Comparisons: For each pair of items, decide which item is more important or preferred based on the established criteria. Mark the preferred item in the corresponding cell of the matrix.
Calculate Scores: After all comparisons are made, count the times each item was preferred. The item with the highest count is ranked highest in priority.

Benefits of Pairwise Comparison

Simplicity: It is easy to understand and implement, requiring no special training[3].
Objectivity: Reduces bias and emotional influence in decision-making by focusing on direct comparisons.
Clarity: Provides a clear ranking of options, making it easier to prioritize tasks or decisions.
Engagement: Encourages collaborative discussions among team members, leading to a better understanding of different perspectives.

Limitations of Pairwise Comparison

Scalability: The number of comparisons increases significantly with the number of items, making it less practical for large lists.
Relative Importance: Does not allow for measuring the intensity of preferences, only the relative ranking.
Cognitive Load: Can be mentally taxing if the list of items is long or the criteria are complex.

Applications of Pairwise Comparison

Project Management: Prioritizing project tasks or deliverables.
Product Development: Ranking features or requirements based on customer needs.
Survey Research: Understanding preferences and establishing relative rankings in surveys.
Strategic Decision-Making: Informing decisions by comparing strategic options or initiatives.

Example of Pairwise Comparison

Imagine a project team needs to prioritize seven project deliverables labeled A to G. They create a pairwise comparison matrix and compare each deliverable against the others. For instance, deliverable A is compared to B, then A to C, and so on. The team marks the preferred deliverable in each comparison. After completing all comparisons, they count the number of times each deliverable was preferred to determine the final ranking.

Comparison of MoSCoW Prioritization, Binary Prioritization, and Pairwise Comparison

Here’s a detailed comparison of the three prioritization methods in a tabular format:

Aspect	MoSCoW Prioritization	Binary Prioritization	Pairwise Comparison
Key Aspects	Categorizes tasks into Must, Should, Could, and Won’t have	Compares requirements in pairs to create a hierarchical list	Compares options in pairs to determine relative preferences
Advantages	Simple to understand, clear categorization, stakeholder alignment	Systematic approach, detailed prioritization, reduces subjectivity	Intuitive, suitable for long lists, provides numerical results
Disadvantages	Subjective categorization, may oversimplify complex projects	Time-consuming for large projects, may become complex	Can be cognitively difficult, potential for inconsistency (transitivity violations)
Clarity	High-level categorization	Detailed prioritization within a hierarchy	Provides clear ranking based on direct comparisons
Stakeholder Involvement	High involvement and alignment required	Less direct involvement, more systematic	Encourages collaborative discussions, but can be intensive
Flexibility	Adaptable to various projects	Best suited for projects with clear requirements	Suitable for both small and large lists, but can be complex for very large sets
Complexity	Simple to understand and implement	More complex and time-consuming	Can be cognitively taxing, especially for large numbers of comparisons
Resource Allocation	Requires careful planning	Systematic but resource-intensive	Requires significant effort for large sets of comparisons

Conclusion

Each prioritization method has its own strengths and weaknesses, making them suitable for different contexts:

MoSCoW Prioritization is ideal for projects needing clear, high-level categorization and strong stakeholder alignment. It is simple and effective for initial prioritization but may lack the granularity needed for more complex projects.
Binary Prioritization offers a systematic and detailed approach, reducing subjectivity. However, it can be time-consuming and complex, especially for large projects.
Pairwise Comparison is intuitive and provides clear numerical results, making it suitable for long lists of options. It encourages collaborative decision-making but can be cognitively challenging and may lead to inconsistencies if not carefully managed.

Choosing the right method depends on the specific needs and context of the decision, including the number of items to prioritize, the level of detail required, and the involvement of stakeholders.

The Lack of Objectivity in Quality Management

ICH Q9(r1) can be reviewed as a revision that addresses long-standing issues of subjectivity in risk management. Subjectivity is a widespread problem throughout the quality sphere, posing significant challenges because it introduces personal biases, emotions, and opinions into decision-making processes that should ideally be driven by objective data and facts.

Inconsistent Decision-Making: Subjective decision-making can lead to inconsistencies because different individuals may have varying opinions and biases. This inconsistency can result in unpredictable outcomes and make it challenging to establish standardized processes. For example, one manager might prioritize customer satisfaction based on personal experiences, while another might focus on cost-cutting, leading to conflicting strategies within the same organization.
Bias and Emotional Influence: Subjectivity often involves emotional influence, which can cloud judgment and lead to decisions not in the organization’s best interest. For instance, a business owner might make decisions based on a personal attachment to a product or service rather than its market performance or profitability. This emotional bias can prevent the business from making necessary changes or investments, ultimately harming its growth and sustainability.
Risk Management Issues: In risk assessments, subjectivity can significantly impact the identification and evaluation of risks. Subjective assessments may overlook critical risks or overemphasize less significant ones, leading to inadequate risk management strategies. Objective, data-driven risk assessments are essential to accurately identify and mitigate potential threats to the business. See ICHQ9(r1).
Difficulty in Measuring Performance: Subjective criteria are often more complicated to quantify and measure, making it challenging to track performance and progress accurately. Objective metrics, such as key performance indicators (KPIs), provide clear, measurable data that can be used to assess the effectiveness of business processes and make informed decisions.
Potential for Misalignment: Subjective decision-making can lead to misalignment between business goals and outcomes. For example, if subjective opinions drive project management decisions, the project may deviate from its original scope, timeline, or budget, resulting in unmet objectives and dissatisfied stakeholders.
Impact on Team Dynamics: Subjectivity can also affect team dynamics and morale. Decisions perceived as biased or unfair can lead to dissatisfaction and conflict among team members. Objective decision-making, based on transparent criteria and data, helps build trust and ensures that all team members are aligned with the business’s goals.

Every organization I’ve been in has a huge problem with subjectivity, and I’m confident in asserting none of us are doing enough to deal with the lack of objectivity, and we mostly rely on our intuition instead of on objective guidelines that will create unambiguous, holistic, and
universally usable models.

Understand the Decisions We Make

Every day, we make many decisions, sometimes without even noticing it. These decisions fall into four categories:

Acceptances: It is a binary choice between accepting or rejecting;
Choices: Opting for a subset from a group of alternatives;
Constructions: Creating an ideal solution given accessible resources;
Evaluations: Here, commitments back up the statements of worth to act

These decisions can be simple or complex, with manifold criteria and several perspectives. Decision-making is the process of choosing an option among manifold alternatives.

The Fallacy of Expert Immunity is a Major Source of Subjectivity

There is a widely incorrect belief that experts are impartial and immune to biases. However, the truth is that no one is immune to bias, not even experts. In many ways, experts are more susceptible to certain biases. The very making of expertise creates and underpins many of the biases. For example, experience and training make experts engage in more selective attention, use chunking and schemas (typical activities and their sequence), and rely on heuristics and expectations arising from past base rate experiences, utilizing a whole range of top-down cognitive processes that create a priori assumptions and expectations.

These cognitive processes often enable experts to make quick and accurate decisions. However, these mechanisms also create bias that can lead them in the wrong direction. Regardless of the utilities (and vulnerability) of such cognitive processing in experts, they do not make experts immune from bias, and indeed, expertise and experience may actually increase (or even cause) certain biases. Experts across domains are subject to cognitive vulnerabilities.

Even when experts are made aware of and acknowledge their biases, they nevertheless think they can overcome them by mere willpower. This is the illusion of control. Combating and countering these biases requires taking specific steps—willpower alone is inadequate to deal with the various manifestations of bias.

In fact, trying to deal with bias through the illusion of control may actually increase the bias due to “ironic processing” or “ironic rebound.” Hence, trying to minimize bias by willpower makes you think of it more and increases its effect. This is similar to a judge instructing jurors to disregard specific evidence. By doing so, the judge makes the jurors notice this evidence even more.

Such fallacies’ beliefs prevent dealing with biases because they dismiss their powers and existence. We need to acknowledge the impact of biases and understand their sources to take appropriate measures when needed and when possible to combat their effects.

Fallacy	Incorrect Belief
Ethical Issues	It only happens to corrupt and unscrupulous individuals, an issue of morals and personal integrity, a question of personal character.
Bad Apples	It only happens to corrupt and unscrupulous individuals. It is an issue of morals and personal integrity, a question of personal character.
Expert Immunity	Experts are impartial and are not affected because bias does not impact competent experts doing their job with integrity.
Technological Protection	Using technology, instrumentation, automation, or artificial intelligence guarantees protection from human biases.
Blind Spot	Other experts are affected by bias, but not me. I am not biased; it is the other experts who are biased.
Illusion of Control	I am aware that bias impacts me, and therefore, I can control and counter its affect. I can overcome bias by mere willpower.

Six Fallacies that Increase Subjectivity

Mitigating Subjectivity

There are four basic strategies to mitigate the impact of subjectivity.

Data-Driven Decision Making

Utilize data and analytics to inform decisions, reducing reliance on personal opinions and biases.

Establish clear metrics with key performance indicators (KPI), key behavior indicators (KBI), and key risk indicators (KRI) that are aligned with objectives.
Implement robust data collection and analysis systems to gather relevant, high-quality data.
Use data visualization tools to present information in an easily digestible format.
Train employees on data literacy and interpretation to ensure proper use of data insights.
Regularly review and update data sources to maintain relevance and accuracy.

Standardized Processes

Implement standardized processes and procedures to ensure consistency and fairness in decision-making.

Document and formalize decision-making procedures across the organization.
Create standardized templates, checklists, and rubrics for evaluating options and making decisions.
Implement a consistent review and approval process for major decisions.
Regularly audit and update standardized processes to ensure they remain effective and relevant.

Education, Training, and Awareness

Educate and train employees and managers on the importance of objective decision-making and recognizing and minimizing personal biases.

Conduct regular training sessions on cognitive biases and their impact on decision-making.
Provide resources and tools to help employees recognize and mitigate their own biases.
Encourage a culture of open discussion and constructive challenge to promote diverse perspectives.
Implement mentoring programs to share knowledge and best practices for objective decision-making.

Digital Tools

Leverage digital tools and software to automate and streamline processes, reducing the potential for subjective influence. The last two is still more aspiration than reality.

Implement workflow management tools to ensure consistent application of standardized processes.
Use collaboration platforms to facilitate transparent and inclusive decision-making processes.
Adopt decision support systems that use algorithms and machine learning to provide recommendations based on data analysis.
Leverage artificial intelligence and predictive analytics to identify patterns and trends that may not be apparent to human decision-makers.

Multi-Criteria Decision-Making to Drive Risk Control

To be honest, too often, we perform a risk assessment not to make decisions but to justify an already existing risk assessment. The risk assessment may help define a few additional action items and determine how rigorous to be about a few things. It actually didn’t make much of an impact on the already-decided path forward. This is some pretty bad risk management and decision-making.

For highly important decisions with high uncertainty or complexity, it is useful to consider the options/alternatives that exist and assess the benefits and risks of each before deciding on a path forward. Thoroughly identifying options/alternatives and assessing the benefits and risks of each can help the decision-making process and ultimately reduce risk.

An effective, highly structured decision-making process can help answer the question, ‘How can we compare the consequences of the various options before deciding?

The most challenging risk decisions are characterized by having several different, important things to consider in an environment where there are often multiple stakeholders and, often, multiple decision-makers.

In Multi-Criteria Decision-Making (MCDM), the primary objective is the structured consideration of the available alternatives (options) for achieving the objectives in order to make the most informed decision, leading to the best outcome.

In a Quality Risk Management context, the decision-making concerns making informed decisions in the face of uncertainty about risks related to the quality (and/or availability) of medicines.

Key Concepts of MCDM

Conflicting Criteria: MCDM deals with situations where criteria conflict. For example, when purchasing a car, one might need to balance cost, comfort, safety, and fuel economy, which often do not align perfectly.
Explicit Evaluation: Unlike intuitive decision-making, MCDM involves a structured approach to explicitly evaluate multiple criteria, which is crucial when the stakes are high, such as deciding whether to build additional manufacturing capacity for a product under development.
Types of Problems:

Multiple-Criteria Evaluation Problems: These involve a finite number of alternatives known at the beginning. The goal is to find the best alternative or a set of good alternatives based on their performance across multiple criteria.
Multiple-Criteria Design Problems: In these problems, alternatives are not explicitly known and must be found by solving a mathematical model. The number of alternatives can be very large, often exponentially.

Preference Information: The methods used in MCDM often require preference information from decision-makers (DMs) to differentiate between solutions. This can be done at various stages of the decision-making process, such as prior articulation of preferences, which transforms the problem into a single-criterion problem.

MCDM focuses on risk and uncertainty by explicitly weighing criteria and trade-offs between them. Multi-criteria decision-making (MCDM) differs from traditional decision-making methods in several key ways:

Explicit Consideration of Multiple Criteria: Traditional decision-making often focuses on a single criterion like cost or profit. MCDM explicitly considers multiple criteria simultaneously, which may be conflicting, such as cost, quality, safety, and environmental impact[1]. This allows for a more comprehensive evaluation of alternatives.
Structured Approach: MCDM provides a structured framework for evaluating alternatives against multiple criteria rather than relying solely on intuition or experience. It involves techniques like weighting criteria, scoring alternatives, and aggregating scores to rank or choose the best option.
Transparency and Consistency: MCDM methods aim to make decision-making more transparent, consistent, and less susceptible to individual biases. The criteria, weights, and evaluation process are explicitly defined, allowing for better justification and reproducibility of decisions.
Quantitative Analysis: Many MCDM methods employ quantitative techniques, such as mathematical models, optimization algorithms, and decision support systems. This enables a more rigorous and analytical approach compared to traditional qualitative methods.
Handling Complexity: MCDM is particularly useful for complex decision problems involving many alternatives, conflicting objectives, and multiple stakeholders. Traditional methods may struggle to handle such complexity effectively.
Stakeholder Involvement: Some MCDM methods, like the Analytic Hierarchy Process (AHP), facilitate the involvement of multiple stakeholders and the incorporation of their preferences and judgments. This can lead to more inclusive and accepted decisions.
Trade-off Analysis: MCDM techniques often involve analyzing trade-offs between criteria, helping decision-makers understand the implications of prioritizing certain criteria over others. This can lead to more informed and balanced decisions.

While traditional decision-making methods rely heavily on experience, intuition, and qualitative assessments, MCDM provides a more structured, analytical, and comprehensive approach, particularly in complex situations with conflicting criteria.

Multi-Criteria Decision-Making (MCDM) is typically performed following these steps:

Define the Decision Problem: Clearly state the problem or decision to be made, identify the stakeholders involved, and determine the desired outcome or objective.
Establish Criteria: Identify the relevant criteria that will be used to evaluate the alternatives. These criteria should be measurable, independent, and aligned with the objectives. Involve stakeholders in selecting and validating the criteria.
Generate Alternatives: Develop a comprehensive list of potential alternatives or options that could solve the problem. Use techniques like brainstorming, benchmarking, or scenario analysis to generate diverse alternatives.
Gather Performance Data: Assess how each alternative performs against each criterion. This may involve quantitative data, expert judgments, or qualitative assessments.
Assign Criteria Weights: By assigning weights, determine each criterion’s relative importance or priority. This can be done through methods like pairwise comparisons, swing weighting, or direct rating. Stakeholder input is crucial here.
Apply MCDM Method: Choose an appropriate MCDM technique based on the problem’s nature and the available data. Some popular methods include: Analytic Hierarchy Process (AHP); Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS); ELimination and Choice Expressing REality (ELECTRE); Preference Ranking Organization METHod for Enrichment of Evaluations (PROMETHEE); and, Multi-Attribute Utility Theory (MAUT).
Evaluate and Rank Alternatives: Apply the chosen MCDM method to evaluate and rank the alternatives based on their performance against the weighted criteria. This may involve mathematical models, software tools, or decision support systems.
Sensitivity Analysis: Perform sensitivity analysis to assess the robustness of the results and understand how changes in criteria weights or performance scores might affect the ranking or choice of alternatives.
Make the Decision: Based on the MCDM analysis, select the most preferred alternative or develop an action plan based on the ranking of alternatives. Involve stakeholders in the final decision-making process.
Monitor and Review: Implement the chosen alternative and monitor its performance. Review the decision periodically, and if necessary, repeat the MCDM process to adapt to changing circumstances or new information.

MCDM is an iterative process; stakeholder involvement, transparency, and clear communication are crucial. Additionally, the specific steps and techniques may vary depending on the problem’s complexity, the data’s availability, and the decision-maker’s preferences.

MCDM Technique	Description	Application	Key Features
Analytic Hierarchy Process (AHP)	A structured technique for organizing and analyzing complex decisions, using mathematics and psychology.	Widely used in business, government, and healthcare for prioritizing and decision-making.	Pairwise comparisons, consistency checks, and hierarchical structuring of criteria and alternatives.
Technique for Order Preference by Similarity to Ideal Solution (TOPSIS)	Based on the concept that the chosen alternative should have the shortest geometric distance from the positive ideal solution and the longest geometric distance from the negative ideal solution.	Frequently used in engineering, management, and human resource management for ranking and selection problems.	Compensatory aggregation, normalization of criteria, and calculation of geometric distances.
Elimination and Choice Expressing Reality (ELECTRE)	An outranking method that compares alternatives by considering both qualitative and quantitative criteria. It uses a pairwise comparison approach to eliminate less favorable alternatives.	Commonly used in project selection, resource allocation, and environmental management.	Use of concordance and discordance indices, handling of both qualitative and quantitative data, and ability to deal with incomplete rankings.
Preference Ranking Organization Method for Enrichment Evaluation (PROMETHEE)	An outranking method that uses preference functions to compare alternatives based on multiple criteria. It provides a complete ranking of alternatives.	Applied in various fields such as logistics, finance, and environmental management.	Preference functions, visual interactive modules (GAIA), and sensitivity analysis.
Multi-Attribute Utility Theory (MAUT)	Involves converting multiple criteria into a single utility function, which is then used to evaluate and rank alternatives. It takes into account the decision-maker’s risk preferences and uncertainties.	Used in complex decision-making scenarios involving risk and uncertainty, such as policy analysis and strategic planning.	Utility functions, probabilistic weights, and handling of uncertainty.

Popular MCDM Techniques

Where I am at

I recently joined Just Evotec Biologics as the Senior Director of Global Quality Engineering and Validation. For a variety of reasons (just look at my past company on my LinkedIn bio and search the news to find one) it was a good time to move. I had decided that I wanted a position that was tied to an innovative manufacturing company and was deep in domain expertise. The combination of Just Evotec Biologics innovative technology aims and the ability to deep dive into one of my favorite topics was just too much to resist. Add to it the opportunity to work with a leader I deeply respected again and well, here I am. And feeling very good about it.

When I first started I met with the team and laid out my 30-60-90 day goals.

As well as talking a little about how I operate.

A big chunk of my time has been getting the lay-of-the-land institutionally. Setting some standards, doing gap assessments, figuring out what-is-what, and getting to know all my partners and stakeholders. For reasons of confidentiality, this post won’t be going deep on that.

What I do want to talk about is our team values and ways of working. I’ve been focused heavily on three areas with the team:

Team Values
Team Decision Making
Team Competencies

Team Values

We did a few workshops where we identified a set of values:

Leader to Team: How I expect the team to perform
Team to Leader: How the Team expects me to perform
Team to Team: How we expect each other to perform

This exercise really helped me understand what was going on within the team and through it I really started to understand some priorities.

For each of these, we created a Value Statement. Here are some examples.

Value: United Front

Definition: Decisions are made and recorded honestly and transparently. Employees understand decisions and how to execute them. The entire team represents the decisions made, and the decision-making process with one voice.

Desired Behaviors:

I hold myself accountable for representing the decisions made by the team.
I work to anticipate and fend off the possibility of failures occurring.
I engage with decision making and respect the decisions that result.

Value: Open to Change

Definition: Willingness to listen to the team. Actively looking for feedback and input from the team before making decisions that impact the team. Open to changing established ways and revisiting previously made decisions.

Desired Behaviors:

I will be transparent with decision-making.
I will create an environment where new ideas are welcome and challenging ideas are encouraged.
I will include the team in decision-making where applicable.
I will actively seek out individual and group feedback to enable continuous improvements.

Value: Learning Culture

Definition: Share lessons learned from projects so team can grow together and remain aligned. Engage in knowledge-sharing sessions.

Desired Behaviors:

I will share lessons learned from each project with the wider QEV team via teams channel &/or weekly team meetings.
I will encourage team members to openly share their experiences, successes, and challenges without fear of judgement.
I will update RAID log with decisions made by the team.
I will identify possible process improvements and update the process improvement tracker

Team Decision Making

Currently working with the team to define decision-making, introducing the RAPID model and working on a matrix of decisions.

Team Competencies

Starting with technical skills we are defining our core competencies. Next, we will tackle, with the larger quality organization, the soft skill side of the equation. This is definitely a work in progress.

Skill Area		Key Aspects	Proficiency Levels
			Beginner	Intermediate	Advanced	Expert
General CQV Principles	� Modern process validation and guidance � Validation design and how to reduce variability		� Able to review a basic protocol � Able to review/approve Validation document deliverables. � Understands the importance of a well-defined URS.	� Able to be QEV lead in a small project � Able to answer questions and guide others in QEV � Participates in process improvement � Able to review and approve RTM/SRs	� Able to be QEV lead in a large project project � Trains and mentors others in QEV � Leads process improvement initiatives � Able to provide Quality oversight on the creation of Validation Plans for complex systems and/or projects	� Sets overall CQV strategy � Recognized as an expert outside of JEB
Facilities and Utilities	� Oversee Facilities, HVAC and Controlled Environments � Pharma Water and WFI � Pure Steam, Compressed Air, Medical Gases		� Understands the principles and GMP requirements	� Applies the principles, activities, and deliverables that constitute an efficient and acceptable approach to demonstrating facility fitness-for-use/qualification	� Guide the Design to Qualification Process for new facilities/utilities or the expansion of existing facilities/utilities	� Able to establish best practices
Systems and Equipment	� Equipment, including Lab equipment		� Understands the principles and GMP requirements	� Principles, activities, and deliverables that constitute an efficient and acceptable approach to demonstrating equipment fitness-for-use/qualification	� Able to provide overall strategy for large projects � Able to be QEV lead on complex systems and equipment.	� Able to establish best practices
Computer Systems and Data Integrity	� Computer lifecycle, including validation		� Understands the principles and GMP requirements	� Able to review CSV documents � Apply GAMP5 risk based approach � Day-to-day quality oversight	� Able to provide overall strategy for a risk based GAMP5 approach to computer system quality	� Able to establish best practices
Asset Lifecycle	� Quality oversight and decision making in the lifecycle asset lifecycle: Plan, acquire, use, maintain, and dispose of assets		� Can use CMMS to look up Calibrations, Cal schedules and PM schedules	� Quality oversight of asset lifecycle decisions � Able to provide oversight on Cal/PM frequency � Able to assess impact to validated state for corrective WO’s.	� Able to establish asset lifecycle for new equipment classes � Establish risk-based PM for new asset classes	� Establish asset lifecycle approach
Quality Systems	� SOP/WI and other GxP Documents � Deviation � Change Control		� Able to use the eQMS	� Deviation reviewer (minor/major) � Change Control approver � Document author/approver	� Deviation reviewer (critical) � Manage umbrella/Parent changes	� Able to set strategic direction
Cleaning, Sanitization and Sterilization Validation	� Evaluate and execute cleaning practices, limit calculations, scientific rationales, and validation documents � Manage the challenges of multi-product facilities in the establishment of limits, determination of validation strategies, and maintaining the validated state		� Differentiate the requirements for cleaning and sterilization validation when using manual, semi-automatic, and automatic cleaning technologies � Review protocols	� Identify and characterize potential residues including product, processing aids, cleaning agents, and adventitious agents � Understand Sterilization principles and requirements � Create, review and approve scientifically sound rationales, validation protocols, and reports	� Manage and remediate the pitfalls inherent in cleaning after the production of biopharmaceutical and pharmaceutical products	� Define cleaning/sterilization validation strategy to meet GMP requirements
Quality Risk Management	� Apply QRM principles according to Q9		� Participate in a risk assessment	� Determine appropriate tools � Establish risk-based decision-making tools	� Set risk-based approaches	� Define risk management program for CQV activities

I’d love feedback on this.

My Overall Philosophy

I’ve been focusing on five key tasks as a leader in this organization:

How I build and gain agreement
Grow the Team
Results and Learning
Deliberate Presence
Prioritizing the Right Relationship

Still a lot to do but I am having a blast.