The Lack of Objectivity in Quality Management

ICH Q9(r1) can be reviewed as a revision that addresses long-standing issues of subjectivity in risk management. Subjectivity is a widespread problem throughout the quality sphere, posing significant challenges because it introduces personal biases, emotions, and opinions into decision-making processes that should ideally be driven by objective data and facts.

Inconsistent Decision-Making: Subjective decision-making can lead to inconsistencies because different individuals may have varying opinions and biases. This inconsistency can result in unpredictable outcomes and make it challenging to establish standardized processes. For example, one manager might prioritize customer satisfaction based on personal experiences, while another might focus on cost-cutting, leading to conflicting strategies within the same organization.
Bias and Emotional Influence: Subjectivity often involves emotional influence, which can cloud judgment and lead to decisions not in the organization’s best interest. For instance, a business owner might make decisions based on a personal attachment to a product or service rather than its market performance or profitability. This emotional bias can prevent the business from making necessary changes or investments, ultimately harming its growth and sustainability.
Risk Management Issues: In risk assessments, subjectivity can significantly impact the identification and evaluation of risks. Subjective assessments may overlook critical risks or overemphasize less significant ones, leading to inadequate risk management strategies. Objective, data-driven risk assessments are essential to accurately identify and mitigate potential threats to the business. See ICHQ9(r1).
Difficulty in Measuring Performance: Subjective criteria are often more complicated to quantify and measure, making it challenging to track performance and progress accurately. Objective metrics, such as key performance indicators (KPIs), provide clear, measurable data that can be used to assess the effectiveness of business processes and make informed decisions.
Potential for Misalignment: Subjective decision-making can lead to misalignment between business goals and outcomes. For example, if subjective opinions drive project management decisions, the project may deviate from its original scope, timeline, or budget, resulting in unmet objectives and dissatisfied stakeholders.
Impact on Team Dynamics: Subjectivity can also affect team dynamics and morale. Decisions perceived as biased or unfair can lead to dissatisfaction and conflict among team members. Objective decision-making, based on transparent criteria and data, helps build trust and ensures that all team members are aligned with the business’s goals.

Every organization I’ve been in has a huge problem with subjectivity, and I’m confident in asserting none of us are doing enough to deal with the lack of objectivity, and we mostly rely on our intuition instead of on objective guidelines that will create unambiguous, holistic, and
universally usable models.

Understand the Decisions We Make

Every day, we make many decisions, sometimes without even noticing it. These decisions fall into four categories:

Acceptances: It is a binary choice between accepting or rejecting;
Choices: Opting for a subset from a group of alternatives;
Constructions: Creating an ideal solution given accessible resources;
Evaluations: Here, commitments back up the statements of worth to act

These decisions can be simple or complex, with manifold criteria and several perspectives. Decision-making is the process of choosing an option among manifold alternatives.

The Fallacy of Expert Immunity is a Major Source of Subjectivity

There is a widely incorrect belief that experts are impartial and immune to biases. However, the truth is that no one is immune to bias, not even experts. In many ways, experts are more susceptible to certain biases. The very making of expertise creates and underpins many of the biases. For example, experience and training make experts engage in more selective attention, use chunking and schemas (typical activities and their sequence), and rely on heuristics and expectations arising from past base rate experiences, utilizing a whole range of top-down cognitive processes that create a priori assumptions and expectations.

These cognitive processes often enable experts to make quick and accurate decisions. However, these mechanisms also create bias that can lead them in the wrong direction. Regardless of the utilities (and vulnerability) of such cognitive processing in experts, they do not make experts immune from bias, and indeed, expertise and experience may actually increase (or even cause) certain biases. Experts across domains are subject to cognitive vulnerabilities.

Even when experts are made aware of and acknowledge their biases, they nevertheless think they can overcome them by mere willpower. This is the illusion of control. Combating and countering these biases requires taking specific steps—willpower alone is inadequate to deal with the various manifestations of bias.

In fact, trying to deal with bias through the illusion of control may actually increase the bias due to “ironic processing” or “ironic rebound.” Hence, trying to minimize bias by willpower makes you think of it more and increases its effect. This is similar to a judge instructing jurors to disregard specific evidence. By doing so, the judge makes the jurors notice this evidence even more.

Such fallacies’ beliefs prevent dealing with biases because they dismiss their powers and existence. We need to acknowledge the impact of biases and understand their sources to take appropriate measures when needed and when possible to combat their effects.

Fallacy	Incorrect Belief
Ethical Issues	It only happens to corrupt and unscrupulous individuals, an issue of morals and personal integrity, a question of personal character.
Bad Apples	It only happens to corrupt and unscrupulous individuals. It is an issue of morals and personal integrity, a question of personal character.
Expert Immunity	Experts are impartial and are not affected because bias does not impact competent experts doing their job with integrity.
Technological Protection	Using technology, instrumentation, automation, or artificial intelligence guarantees protection from human biases.
Blind Spot	Other experts are affected by bias, but not me. I am not biased; it is the other experts who are biased.
Illusion of Control	I am aware that bias impacts me, and therefore, I can control and counter its affect. I can overcome bias by mere willpower.

Six Fallacies that Increase Subjectivity

Mitigating Subjectivity

There are four basic strategies to mitigate the impact of subjectivity.

Data-Driven Decision Making

Utilize data and analytics to inform decisions, reducing reliance on personal opinions and biases.

Establish clear metrics with key performance indicators (KPI), key behavior indicators (KBI), and key risk indicators (KRI) that are aligned with objectives.
Implement robust data collection and analysis systems to gather relevant, high-quality data.
Use data visualization tools to present information in an easily digestible format.
Train employees on data literacy and interpretation to ensure proper use of data insights.
Regularly review and update data sources to maintain relevance and accuracy.

Standardized Processes

Implement standardized processes and procedures to ensure consistency and fairness in decision-making.

Document and formalize decision-making procedures across the organization.
Create standardized templates, checklists, and rubrics for evaluating options and making decisions.
Implement a consistent review and approval process for major decisions.
Regularly audit and update standardized processes to ensure they remain effective and relevant.

Education, Training, and Awareness

Educate and train employees and managers on the importance of objective decision-making and recognizing and minimizing personal biases.

Conduct regular training sessions on cognitive biases and their impact on decision-making.
Provide resources and tools to help employees recognize and mitigate their own biases.
Encourage a culture of open discussion and constructive challenge to promote diverse perspectives.
Implement mentoring programs to share knowledge and best practices for objective decision-making.

Digital Tools

Leverage digital tools and software to automate and streamline processes, reducing the potential for subjective influence. The last two is still more aspiration than reality.

Implement workflow management tools to ensure consistent application of standardized processes.
Use collaboration platforms to facilitate transparent and inclusive decision-making processes.
Adopt decision support systems that use algorithms and machine learning to provide recommendations based on data analysis.
Leverage artificial intelligence and predictive analytics to identify patterns and trends that may not be apparent to human decision-makers.

Multi-Criteria Decision-Making to Drive Risk Control

To be honest, too often, we perform a risk assessment not to make decisions but to justify an already existing risk assessment. The risk assessment may help define a few additional action items and determine how rigorous to be about a few things. It actually didn’t make much of an impact on the already-decided path forward. This is some pretty bad risk management and decision-making.

For highly important decisions with high uncertainty or complexity, it is useful to consider the options/alternatives that exist and assess the benefits and risks of each before deciding on a path forward. Thoroughly identifying options/alternatives and assessing the benefits and risks of each can help the decision-making process and ultimately reduce risk.

An effective, highly structured decision-making process can help answer the question, ‘How can we compare the consequences of the various options before deciding?

The most challenging risk decisions are characterized by having several different, important things to consider in an environment where there are often multiple stakeholders and, often, multiple decision-makers.

In Multi-Criteria Decision-Making (MCDM), the primary objective is the structured consideration of the available alternatives (options) for achieving the objectives in order to make the most informed decision, leading to the best outcome.

In a Quality Risk Management context, the decision-making concerns making informed decisions in the face of uncertainty about risks related to the quality (and/or availability) of medicines.

Key Concepts of MCDM

Conflicting Criteria: MCDM deals with situations where criteria conflict. For example, when purchasing a car, one might need to balance cost, comfort, safety, and fuel economy, which often do not align perfectly.
Explicit Evaluation: Unlike intuitive decision-making, MCDM involves a structured approach to explicitly evaluate multiple criteria, which is crucial when the stakes are high, such as deciding whether to build additional manufacturing capacity for a product under development.
Types of Problems:

Multiple-Criteria Evaluation Problems: These involve a finite number of alternatives known at the beginning. The goal is to find the best alternative or a set of good alternatives based on their performance across multiple criteria.
Multiple-Criteria Design Problems: In these problems, alternatives are not explicitly known and must be found by solving a mathematical model. The number of alternatives can be very large, often exponentially.

Preference Information: The methods used in MCDM often require preference information from decision-makers (DMs) to differentiate between solutions. This can be done at various stages of the decision-making process, such as prior articulation of preferences, which transforms the problem into a single-criterion problem.

MCDM focuses on risk and uncertainty by explicitly weighing criteria and trade-offs between them. Multi-criteria decision-making (MCDM) differs from traditional decision-making methods in several key ways:

Explicit Consideration of Multiple Criteria: Traditional decision-making often focuses on a single criterion like cost or profit. MCDM explicitly considers multiple criteria simultaneously, which may be conflicting, such as cost, quality, safety, and environmental impact[1]. This allows for a more comprehensive evaluation of alternatives.
Structured Approach: MCDM provides a structured framework for evaluating alternatives against multiple criteria rather than relying solely on intuition or experience. It involves techniques like weighting criteria, scoring alternatives, and aggregating scores to rank or choose the best option.
Transparency and Consistency: MCDM methods aim to make decision-making more transparent, consistent, and less susceptible to individual biases. The criteria, weights, and evaluation process are explicitly defined, allowing for better justification and reproducibility of decisions.
Quantitative Analysis: Many MCDM methods employ quantitative techniques, such as mathematical models, optimization algorithms, and decision support systems. This enables a more rigorous and analytical approach compared to traditional qualitative methods.
Handling Complexity: MCDM is particularly useful for complex decision problems involving many alternatives, conflicting objectives, and multiple stakeholders. Traditional methods may struggle to handle such complexity effectively.
Stakeholder Involvement: Some MCDM methods, like the Analytic Hierarchy Process (AHP), facilitate the involvement of multiple stakeholders and the incorporation of their preferences and judgments. This can lead to more inclusive and accepted decisions.
Trade-off Analysis: MCDM techniques often involve analyzing trade-offs between criteria, helping decision-makers understand the implications of prioritizing certain criteria over others. This can lead to more informed and balanced decisions.

While traditional decision-making methods rely heavily on experience, intuition, and qualitative assessments, MCDM provides a more structured, analytical, and comprehensive approach, particularly in complex situations with conflicting criteria.

Multi-Criteria Decision-Making (MCDM) is typically performed following these steps:

Define the Decision Problem: Clearly state the problem or decision to be made, identify the stakeholders involved, and determine the desired outcome or objective.
Establish Criteria: Identify the relevant criteria that will be used to evaluate the alternatives. These criteria should be measurable, independent, and aligned with the objectives. Involve stakeholders in selecting and validating the criteria.
Generate Alternatives: Develop a comprehensive list of potential alternatives or options that could solve the problem. Use techniques like brainstorming, benchmarking, or scenario analysis to generate diverse alternatives.
Gather Performance Data: Assess how each alternative performs against each criterion. This may involve quantitative data, expert judgments, or qualitative assessments.
Assign Criteria Weights: By assigning weights, determine each criterion’s relative importance or priority. This can be done through methods like pairwise comparisons, swing weighting, or direct rating. Stakeholder input is crucial here.
Apply MCDM Method: Choose an appropriate MCDM technique based on the problem’s nature and the available data. Some popular methods include: Analytic Hierarchy Process (AHP); Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS); ELimination and Choice Expressing REality (ELECTRE); Preference Ranking Organization METHod for Enrichment of Evaluations (PROMETHEE); and, Multi-Attribute Utility Theory (MAUT).
Evaluate and Rank Alternatives: Apply the chosen MCDM method to evaluate and rank the alternatives based on their performance against the weighted criteria. This may involve mathematical models, software tools, or decision support systems.
Sensitivity Analysis: Perform sensitivity analysis to assess the robustness of the results and understand how changes in criteria weights or performance scores might affect the ranking or choice of alternatives.
Make the Decision: Based on the MCDM analysis, select the most preferred alternative or develop an action plan based on the ranking of alternatives. Involve stakeholders in the final decision-making process.
Monitor and Review: Implement the chosen alternative and monitor its performance. Review the decision periodically, and if necessary, repeat the MCDM process to adapt to changing circumstances or new information.

MCDM is an iterative process; stakeholder involvement, transparency, and clear communication are crucial. Additionally, the specific steps and techniques may vary depending on the problem’s complexity, the data’s availability, and the decision-maker’s preferences.

MCDM Technique	Description	Application	Key Features
Analytic Hierarchy Process (AHP)	A structured technique for organizing and analyzing complex decisions, using mathematics and psychology.	Widely used in business, government, and healthcare for prioritizing and decision-making.	Pairwise comparisons, consistency checks, and hierarchical structuring of criteria and alternatives.
Technique for Order Preference by Similarity to Ideal Solution (TOPSIS)	Based on the concept that the chosen alternative should have the shortest geometric distance from the positive ideal solution and the longest geometric distance from the negative ideal solution.	Frequently used in engineering, management, and human resource management for ranking and selection problems.	Compensatory aggregation, normalization of criteria, and calculation of geometric distances.
Elimination and Choice Expressing Reality (ELECTRE)	An outranking method that compares alternatives by considering both qualitative and quantitative criteria. It uses a pairwise comparison approach to eliminate less favorable alternatives.	Commonly used in project selection, resource allocation, and environmental management.	Use of concordance and discordance indices, handling of both qualitative and quantitative data, and ability to deal with incomplete rankings.
Preference Ranking Organization Method for Enrichment Evaluation (PROMETHEE)	An outranking method that uses preference functions to compare alternatives based on multiple criteria. It provides a complete ranking of alternatives.	Applied in various fields such as logistics, finance, and environmental management.	Preference functions, visual interactive modules (GAIA), and sensitivity analysis.
Multi-Attribute Utility Theory (MAUT)	Involves converting multiple criteria into a single utility function, which is then used to evaluate and rank alternatives. It takes into account the decision-maker’s risk preferences and uncertainties.	Used in complex decision-making scenarios involving risk and uncertainty, such as policy analysis and strategic planning.	Utility functions, probabilistic weights, and handling of uncertainty.

Popular MCDM Techniques

Building the Risk Team

Good risk assessments are a team effort. If done right this is a key way to reduce subjectivity and it recognizes that none of us know everything.

An effective risk team:

Has expertise in the subject
Knows how to drive decision making
Includes representatives of all relevant stakeholders
Is cross-functional and multidisciplinary
Allows appropriate and applicable empowerment

One of the core jobs of a process owner in risk assessment is assembling this team and ensuring they have the space to do their job. They are often called the champion or sponsor for good reason.

It is important to keep in mind that membership of this team will change, gaining and losing members and bringing on people for specific subsections, depending on the scale and scope of the risk assessment.

The more complex the scope and the more involved the assessment tool, the more important it is to have a facilitator to drive the process. This allows someone to focus on the process of the risk assessment, and the reduction of subjectivity.

Quality, Decision Making and Putting the Human First

Quality stands in a position, sometimes uniquely in an organization, of engaging with stakeholders to understand what objectives and unique positions the organization needs to assume, and the choices that are making in order to achieve such objectives and positions.

The effectiveness of the team in making good decisions by picking the right choices depends on their ability of analyzing a problem and generating alternatives. As I discussed in my post “Design Lifecycle within PDCA – Planning” experimentation plays a critical part of the decision making process. When designing the solution we always consider:

Always include a “do nothing” option: Not every decision or problem demands an action. Sometimes, the best way is to do nothing.
How do you know what you think you know? This should be a question everyone is comfortable asking. It allows people to check assumptions and to question claims that, while convenient, are not based on any kind of data, firsthand knowledge, or research.
Ask tough questions! Be direct and honest. Push hard to get to the core of what the options look like.
Have a dissenting option. It is critical to include unpopular but reasonable options. Make sure to include opinions or choices you personally don’t like, but for which good arguments can be made. This keeps you honest and gives anyone who see the pros/cons list a chance to convince you into making a better decision than the one you might have arrived at on your own.
Consider hybrid choices. Sometimes it’s possible to take an attribute of one choice and add it to another. Like exploratory design, there are always interesting combinations in decision making. This can explode the number of choices, which can slow things down and create more complexity than you need. Watch for the zone of indifference (options that are not perceived as making any difference or adding any value) and don’t waste time in it.
Include all relevant perspectives. Consider if this decision impacts more than just the area the problem is identified in. How does it impact other processes? Systems?

A struggle every organization has is how to think through problems in a truly innovative way. Installing new processes into an old bureaucracy will only replace one form of control with another. We need to rethink the very matter of control and what it looks like within an organization. It is not about change management, on it sown change management will just shift the patterns of the past. To truly transform we need a new way of thinking.

One of my favorite books on just how to do this is Humanocracy: Creating Organizations as Amazing as the People Inside Them by Gary Hamel and Michele Zanini. In this book, the authors advocate that business must become more fundamentally human first. The idea of human ability and how to cultivate and unleash it is an underlying premise of this book.

it’s possible to capture the benefits of bureaucracy—control, consistency, and coordination—while avoiding the penalties—inflexibility, mediocrity, and apathy.
Gary Hamel and Michele Zanini, Humanocracy, p. 15

The above quote really encapsulates the heart of this book, and why I think it is such a pivotal read for my peers. This books takes the core question of a bureaurcacy is “How do we get human beings to better serve the organization?”. The issue at the heart of humanocracy becomes: “What sort of organization elicits and merits the best that human beings can give?” Seems a simple swap, but the implications are profound.

Bureaucracy versus Humanocracy. Source: Gary Hamel and Michele Zanini, *Humanocracy*, p. 48

I would hope you, like me, see the promise of many of the central tenets of Quality Management, not least Deming’s 8th point. The very real tendency of quality to devolve to pointless bureaucracy is something we should always be looking to combat.

Humanocracy’s central point is that by truly putting the employee first in our organizations we drive a human-centered organization that powers and thrives on innovation. Humanocracy is particularly relevant as organizations seek to be more resilient, agile, adaptive, innovative, customer centric etc. Leaders pursuing such goals seek to install systems like agile, devops, flexible teams etc. They will fail, because people are not processes. Resiliency, agility, efficiency, are not new programming codes for people. These goals require more than new rules or a corporate initiative. Agility, resilience, etc. are behaviors, attitudes, ways of thinking that can only work when you change the deep ‘systems and assumptions’ within an organization. This book discusses those deeper changes.

Humanocracy lays out seven tips for success in experimentation. I find they align nicely with Kotter’s 8 change accelerators.

Humanocracy’s Tip	Kotter’s Accelerator
Keep it Simple	Generate (and celebrate) short-term wins
Use Volunteers	Enlist a volunteer army
Make it Fun	Sustain Acceleration
Start in your own backyard	Form a change vision and strategic initiatives
Run the new parallel with the old	Enable action by removing barriers
Refine and Retest	Sustain acceleration
Stay loyal to the problem	Create a Sense of Urgency around a Big Opportunity

Comparison to Kotter’s Eight Accelerators for Change

Teams reason better

Teams collaborate better than individuals on a wide range of problem-solving for two reason:

People are exposed to points of view different from their own. If the arguments are good enough, people can change their mind to adopt better beliefs. This requires structure, such as “Yes…but…and“
The back-and-forth of a conversation allows people to address counterarguments, and thus to refine their arguments, making it more likely that the best argument carries the day.

Both of these work to reduce bias and subjectivity.

There are a few principles to make this team collaboration work.

Clear purpose: What is the reason for the collaboration? What’s the business case or business need? Without alignment on the purpose and its underlying importance to the organization, the collaboration will fail. The scope will start to change, or other priorities will take precedence.
Clear process: How will the collaboration take place? What are the steps? What is the timing? Who is responsible for what?
Clear expectations: What is the specific goal or outcome we are striving for through this collaboration?
Clear support: Problems will arise that the team cannot handle on their own. In those cases, what is the escalation process, including who and when?

Ensure these are in your team ground rules, measure success and perform continuous improvement.