Overcoming Subjectivity in Risk Management and Decision Making Requires a Culture of Quality and Excellence

Risk assessments, problem solving and making good decisions need teams, but any team has challenges in group think it must overcome. Ensuring your facilitators, team leaders and sponsors are aware and trained on these biases will help lead to deal with subjectivity, understand uncertainty and drive to better outcomes. But no matter how much work you do there, it won’t make enough of a difference until you’ve built a culture of quality and excellence.

The mindsets we are trying to build into our culture will strive to overcome a few biases in our teams that lead to subjectivity.

Bias Toward Fitting In

We have a natural desire to want to fit in. This tendency leads to two challenges:

Challenge #1: Believing we need to conform. Early in life, we realize that there are tangible benefits to be gained from following social and organizational norms and rules. As a result, we make a significant effort to learn and adhere to written and unwritten codes of behavior at work. But here’s the catch: Doing so limits what we bring to the organization.

Challenge #2: Failure to use one’s strengths. When employees conform to what they think the organization wants, they are less likely to be themselves and to draw on their strengths. When people feel free to stand apart from the crowd, they can exercise their signature strengths (such as curiosity, love for learning, and perseverance), identify opportunities for improvement, and suggest ways to exploit them. But all too often, individuals are afraid of rocking the boat.

We need to use several methods to combat the bias toward fitting in. These need to start at the cultural level. Risk management, problem solving and decision making only overcome biases when embedded in a wider, effective culture.

Encourage people to cultivate their strengths. To motivate and support employees, some companies allow them to spend a certain portion of their time doing work of their own choosing. Although this is a great idea, we need to build our organization to help individuals apply their strengths every day as a normal part of their jobs.

Managers need to help individuals identify and develop their fortes—and not just by discussing them in annual performance reviews. Annual performance reviews are horribly ineffective. Just by using “appreciation jolt”, positive feedback., can start to improve the culture. It’s particularly potent when friends, family, mentors, and coworkers share stories about how the person excels. These stories trigger positive emotions, cause us to realize the impact that we have on others, and make us more likely to continue capitalizing on our signature strengths rather than just trying to fit in.

Managers should ask themselves the following questions: Do I know what my employees’ talents and passions are? Am I talking to them about what they do well and where they can improve? Do our goals and objectives include making maximum use of employees’ strengths?

Increase awareness and engage workers. If people don’t see an issue, you can’t expect them to speak up about it.

Model good behavior. Employees take their cues from the managers who lead them.

Bias Toward Experts

This is going to sound counter-intuitive, especially since expertise is so critical. Yet our biases about experts can cause a few challenges.

Challenge #1: An overly narrow view of expertise. Organizations tend to define “expert” too narrowly, relying on indicators such as titles, degrees, and years of experience. However, experience is a multidimensional construct. Different types of experience—including time spent on the front line, with a customer or working with particular people—contribute to understanding a problem in detail and creating a solution.

A bias toward experts can also lead people to misunderstand the potential drawbacks that come with increased time and practice in the job. Though experience improves efficiency and effectiveness, it can also make people more resistant to change and more likely to dismiss information that conflicts with their views.

Challenge #2: Inadequate frontline involvement. Frontline employees—the people directly involved in creating, selling, delivering, and servicing offerings and interacting with customers—are frequently in the best position to spot and solve problems. Too often, though, they aren’t empowered to do so.

The following tactics can help organizations overcome weaknesses of the expert bias.

Encourage workers to own problems that affect them. Make sure that your organization is adhering to the principle that the person who experiences a problem should fix it when and where it occurs. This prevents workers from relying too heavily on experts and helps them avoid making the same mistakes again. Tackling the problem immediately, when the relevant information is still fresh, increases the chances that it will be successfully resolved. Build a culture rich with problem-solving and risk management skills and behaviors.

Give workers different kinds of experience. Recognize that both doing the same task repeatedly (“specialized experience”) and switching between different tasks (“varied experience”) have benefits. Yes, Over the course of a single day, a specialized approach is usually fastest. But over time, switching activities across days promotes learning and kept workers more engaged. Both specialization and variety are important to continuous learning.

Empower employees to use their experience. Organizations should aggressively seek to identify and remove barriers that prevent individuals from using their expertise. Solving the customer’s problems in innovative, value-creating ways—not navigating organizational impediments— should be the challenging part of one’s job.

In short we need to build the capability to leverage all level of experts, and not just a few in their ivory tower.

These two biases can be overcome and through that we can start building the mindsets to deal effectively with subjectivity and uncertainty. Going further, build the following as part of our team activities as sort of a quality control checklist:

Check for self-interest bias
Check for the affect heuristic. Has the team fallen in love with its own output?
Check for group think. Were dissenting views explored adequately?
Check for saliency bias. Is this routed in past successes?
Check for confirmation bias.
Check for availability bias
Check for anchoring bias
Check for halo effect
Check for sunk cost fallacy and endowment effect
Check for overconfidence, planning fallacy, optimistic biases, competitor neglect
Check for disaster neglect. Have the team conduct a post-mortem: Imagine that the worst has happened and develop a story about its causes.
Check for loss aversion

Uncertainty and Subjectivity in Risk Management

The July-2019 monthly gift to members of the ASQ is a lot of material on Failure Mode and Effect Analysis (FMEA). Reading through the material got me to thinking of subjectivity in risk management.

Risk assessments have a core of the subjective to them, frequently including assumptions about the nature of the hazard, possible exposure pathways, and judgments for the likelihood that alternative risk scenarios might occur. Gaps in the data and information about hazards, uncertainty about the most likely projection of risk, and incomplete understanding of possible scenarios contribute to uncertainties in risk assessment and risk management. You can go even further and say that risk is socially constructed, and that risk is at once both objectively verifiable and what we perceive or feel it to be. Then again, the same can be said of most of science.

Risk is a future chance of loss given exposure to a hazard. Risk estimates, or qualitative ratings of risk, are necessarily projections of future consequences. Thus, the true probability of the risk event and its consequences cannot be known in advance. This creates a need for subjective judgments to fill-in information about an uncertain future. In this way risk management is rightly seen as a form of decision analysis, a form of making decisions against uncertainty.

Everyone has a mental picture of risk, but the formal mathematics of risk analysis are inaccessible to most, relying on probability theory with two major schools of thought: the frequency school and the subjective probability school. The frequency school says probability is based on a count of the number of successes divided by total number of trials. Uncertainty that is ready characterized using frequentist probability methods is “aleatory” – due to randomness (or random sampling in practice). Frequentist methods give an estimate of “measured” uncertainty; however, it is arguably trapped in the past because it does not lend itself to easily to predicting future successes.

In risk management we tend to measure uncertainty with a combination of frequentist and subjectivist probability distributions. For example, a manufacturing process risk assessment might begin with classical statistical control data and analyses. But projecting the risks from a process change might call for expert judgments of e.g. possible failure modes and the probability that failures might occur during a defined period. The risk assessor(s) bring prior expert knowledge and, if we are lucky, some prior data, and start to focus the target of the risk decision using subjective judgments of probabilities.

Some have argued that a failure to formally control subjectivity — in relation to probability judgments – is the failure of risk management. This was an argument that some made during WCQI, for example. Subjectivity cannot be eliminated nor is it an inherent limitation. Rather, the “problem with subjectivity” more precisely concerns two elements:

A failure to recognize where and when subjectivity enters and might create problems in risk assessment and risk-based decision making; and
A failure to implement controls on subjectivity where it is known to occur.

Risk is about the chance of adverse outcomes of events that are yet to occur, subjective judgments of one form or another will always be required in both risk assessment and risk management decision-making.

We control subjectivity in risk management by:

Raising awareness of where/when subjective judgments of probability occur in risk assessment and risk management
Identifying heuristics and biases where they occur
Improving the understanding of probability among the team and individual experts
Calibrating experts individually
Applying knowledge from formal expert elicitation
Use expert group facilitation when group probability judgments are sought

Each one of these is it’s own, future, post.

WCQI Day 3 – morning

I didn’t make it to the key note. I had a work conference call so I will never learn the quality secrets of Anheuser-Busch.

“A Fresh Approach to Risk Assessment & FMEA: It’s all about severity” by Beverly Daniels.

After yesterday’s Quality 4.0 session I was not going to miss this as the presenter has a blunt, to the point attitutde, that could be interesting and fun to watch.

Very R&R driven mindset, which is a little far away for me but one I find fascinating. Her approach is to get rid of probability and detection on an FMEA. How does she do that?

Create a function diagram and process maps as applicable
Create an input:output matrix
List functions
List failure modes: how a failure presents itself
List the effects of the failure modes
Determine severity of the failure modes at the local level and system level
Develop V&V, mitigation and control plans for all high severity failures.

Which means she’s just not using the risk assessment as a consolidation of decisions (hopefully using some other form of matrix) and always uses testing data for occurrence.

The speaker made the point about static FMEA’s a lot, I’m a big fan of living risk assessments, and I think that is an approach that needs more attention.

Some interesting ideas on probability and testing here, but buried under some strong rhetoric. Luckily she posted a longer write-up which I’ll need to consider more.

“Using Decision Analysis to Improve, Make or Break Decisions” by Kurt Stuke

Someday I’ll write-up more on why I find long credential porn intros annoying. My favorite intro is “Jeremiah Genest works for Sanofi and has 20 years of experience in quality.” Post my damn CV if you want, but seriously my words, my presentation and my references should speak for themselves.

I like the flip sessions, prepping prior is always good. The conference needs to do a better job letting people know about the prep work. The amount of confusion in this session was telling. The app does not even link to the prep work, only way is an email.

Here are Kurt’s resources: https://www.kurtstuke.com/OER/WCQI/

There is no 100% tool, glad he stresses that at the beginning, as we sometimes forget to do that in the profession.

“Whim leads to advocacy approach which means data looses its voice.”

Used KT as a way for decision analysis. Talking about the “must haves” and “nice-to-haves” Maybe it’s because of the proprietary nature of KT, but I feel their methodology is either someone folks are really familiar with or surprised by.

So this is again basic stuff. I’m not sure if this is what I am deciding to go to or if just where I am in my journey. At my table I was the only one really familiar with these tools.

Good presenter. Love the workshop approach. It was great watching and participating with my table-mates and seeing lightbulbs go off. However, this is a basic workshop and not intermediate.

Risk Management is about reducing uncertainty

Risk Management is all about eliminating surprise. So to truly start to understand our risks, we need to understand uncertainty, we need to understand the unknowns. Borrowing from Andreas Schamanek’s Taxonomies of the unknown, let’s explore a few of the various taxonomies of what is not known.

Ignorance Map

I’m pretty sure Ann Kerwin first gave us the “known unknowns” and the “unknown knowns” that people still find a source of amusement about former defense secretary Rumsfield.

	Known	Unknown
Known	Known knowns	Known unknowns (conscious ignorance)
Unknown	Unknown knowns (tacit knowledge)	Unknown unknowns (meta-ignorance)

Understanding uncertainty involves knowledge management, this is why a rigorous knowledge management program is a prerequisite for an effective quality management system.

Risk management is then a way of teasing out the unknowns and allowing us to take action:

Risk assessments mostly easily focus on the ignorance that we are aware of, the ‘known unknowns’.
Risk assessments can also serve as a tool of teasing out the ‘unknown knowns’. This is why participation of subject matter experts is so critical. Through the formal methodology of the risk assessment we expose and explore tacit knowledge.
The third kind of ignorance is what we do now know we do not know, the ‘unknown unknowns’. We generally become aware of unknown unknowns in two ways: hindsight (deviations) and by purposefully expanding our horizons. This expansion includes diversity and also good experimentation. It is the hardest, but perhaps, most valuable part of risk management.

Taxonomy of Ignorance

Different Kinds of Unknowns, Source: Smithson (1989, p. 9); also in Bammer et al. (2008, p. 294).

Smithson distinguishes between passive and active ignorance. Passive ignorance involves areas that we are ignorant of, whereas active ignorance refers to areas we ignore. He uses the term ‘error’ for the unknowns encompassed by passive ignorance and ‘irrelevance’ for active ignorance.

Taboo is fascinating because it gets to the heart of our cultural blindness, those parts of our organization that are closed to scrutiny.

Smithson can help us understand why risk assessments are both a qualitative and a quantitative endeavor. While dealing with the unknown is the bread and butter of statistics, only a small part of the terrain of uncertainty is covered. Under Smithson’s typology, statistics primarily operates in the area of incompleteness, across probability and some kinds of vagueness. In terms of its considerations of sampling bias, statistics also has some overlap with inaccuracy. But, as the typology shows, there is much more to unknowns than the areas statistics deals with. This is another reason that subject matter experts, and different ways of thinking is a must.

Ensuring wide and appropriate expert participation gives additional perspectives on unknowns. There is also synergies by finding unrecognized similarities between disciplines and stakeholders in the unknowns they deal with and there may be great benefit from combining forces. It is important to use these concerns to enrich thinking about unknowns, rather than ruling them out as irrelevant.

Sources of Surprise

Risk management is all about managing surprise. It helps to break surprise down to three types: risk, uncertainty and ignorance.

Risk: The condition in which the event, process, or outcomes and the probability that each will occur is known.
- Issue: In reality, complete knowledge of probabilities and range of potential outcomes or consequences is not usually known and is sometimes unknowable.
Uncertainty: The condition in which the event, process, or outcome is known (factually or hypothetically) but the probabilities that it will occur are not known.
- Issue: The probabilities assigned, if any, are subjective, and ways to establish reliability for different subjective probability estimates are debatable.
Ignorance: The condition in which the event, process, or outcome is not known or expected.
- Issue: How can we anticipate the unknown, improve the chances of anticipating, and, therefore, improve the chances of reducing vulnerability?

Effective use of the methodology moves ideally from ignorance to eventually risk.

Ignorance	Description	Methods of Mitigation
Closed Ignorance	Information is available but SMEs are unwilling or unable to consider that some outcomes are unknown to them.	Self-audit process, regular third-party audits, and open and transparent system with global participation
Open Ignorance	Information is available and SMEs are willing to recognize and consider that some outcomes are unknown.
Personal	Surprise occurs because an individual SME lacks knowledge or awareness of the available information.	effective teams xxplore multiple perspectives by including a diverse set of individuals and data sources for data gathering and analysis. Transparency in process.
Communal	Surprise occurs because a group of SMEs has only similar viewpoints represented or may be less willing to consider views outside the community.	Diversity of viewpoints and sue of tools to overcome group-think and “tribal” knowledge
Novelty	Surprise occurs because the SMEs are unable to anticipate and prepare for external shocks or internal changes in preferences, technologies, and institutions.	Simulating impacts and gaming alternative outcomes of various potentials under different conditions (Blue Team/Read Team exercises)
Complexity	Surprise occurs when inadequate forecasting tools are used to analyze the available data, resulting in inter-relationships, hidden dependencies, feedback loops, and other negative factors that lead to inadequate or incomplete understanding of the data.	System Thinking Track changes and interrelationships of various systems to discover potential macro-effect force changes 12-Levers

Risk Management is all about understanding surprise and working to reduce uncertainty and ignorance in order to reduce, eliminate and sometimes accept. As a methodology it is effective at avoiding surrender and denial. With innovation we can even contemplate exploitation. As organizations mature, it is important to understand these concepts and utilize them.

References

Gigerenzer, Gerd and Garcia-Retamero, Rocio. Cassandra’s Regret: The Psychology of Not Wanting to Know (March 2017), Psychological Review, 2017, Vol. 124, No. 2, 179–196.
House, Robert J., Paul J. Hanges, Mansour Javidan, Peter Dorfman, and Vipin Gupta, eds. 2004. Culture, Leadership, and Organizations: The GLOBE Study of 62 Societies. Thousand Oaks, Calif.: Sage Publications.
Kerwin, A. (1993). None Too Solid: Medical Ignorance. Knowledge, 15(2), 166–185.
Smithson, M. (1989) Ignorance and Uncertainty: Emerging Paradigms, New York: Springer-Verlag.
Smithson, M. (1993) “Ignorance and Science”, Knowledge: Creation, Diffusion, Utilization, 15(2) December: 133-156.

Risk Management of Raw Materials

This paper discusses background information related to RM regulatory requirements and industry challenges, and then highlights key principles to consider in setting up a risk-based RM management approach and control strategy. This paper then provides an example of how to translate those key principles into a detailed RM risk assessment methodology, and how to apply this methodology to specific raw materials. To better illustrate the diversity and nuance in applying a corresponding RM control strategy, a number of case studies with raw materials typically utilized in the manufacture of biological medicinal products have been included as well as discussion on phase-based mitigations.
European Biopharmaceutical Enterprises (2018) “Management and Control of Raw Materials Used in the Manufacture of Biological Medicinal Products and ATMPs“

Good foundation document for how to build a risk management program for managing raw materials.