A concept map of Churchman’s general systems approach.
— Read on csl4d.wordpress.com/2019/07/27/a-concept-map-of-churchmans-general-systems-approach/
Great overview of Churchman’s approach. Highly recommended reading.
Category: system lifecycle
Uncertainty and Subjectivity in Risk Management
The July-2019 monthly gift to members of the ASQ is a lot of material on Failure Mode and Effect Analysis (FMEA). Reading through the material got me to thinking of subjectivity in risk management.
Risk assessments have a core of the subjective to them, frequently including assumptions about the nature of the hazard, possible exposure pathways, and judgments for the likelihood that alternative risk scenarios might occur. Gaps in the data and information about hazards, uncertainty about the most likely projection of risk, and incomplete understanding of possible scenarios contribute to uncertainties in risk assessment and risk management. You can go even further and say that risk is socially constructed, and that risk is at once both objectively verifiable and what we perceive or feel it to be. Then again, the same can be said of most of science.
Risk is a future chance of loss given exposure to a hazard. Risk estimates, or qualitative ratings of risk, are necessarily projections of future consequences. Thus, the true probability of the risk event and its consequences cannot be known in advance. This creates a need for subjective judgments to fill-in information about an uncertain future. In this way risk management is rightly seen as a form of decision analysis, a form of making decisions against uncertainty.
Everyone has a mental picture of risk, but the formal mathematics of risk analysis are inaccessible to most, relying on probability theory with two major schools of thought: the frequency school and the subjective probability school. The frequency school says probability is based on a count of the number of successes divided by total number of trials. Uncertainty that is ready characterized using frequentist probability methods is “aleatory” – due to randomness (or random sampling in practice). Frequentist methods give an estimate of “measured” uncertainty; however, it is arguably trapped in the past because it does not lend itself to easily to predicting future successes.
In risk management we tend to measure uncertainty with a combination of frequentist and subjectivist probability distributions. For example, a manufacturing process risk assessment might begin with classical statistical control data and analyses. But projecting the risks from a process change might call for expert judgments of e.g. possible failure modes and the probability that failures might occur during a defined period. The risk assessor(s) bring prior expert knowledge and, if we are lucky, some prior data, and start to focus the target of the risk decision using subjective judgments of probabilities.
Some have argued that a failure to formally control subjectivity — in relation to probability judgments – is the failure of risk management. This was an argument that some made during WCQI, for example. Subjectivity cannot be eliminated nor is it an inherent limitation. Rather, the “problem with subjectivity” more precisely concerns two elements:
- A failure to recognize where and when subjectivity enters and might create problems in risk assessment and risk-based decision making; and
- A failure to implement controls on subjectivity where it is known to occur.
Risk is about the chance of adverse outcomes of events that are yet to occur, subjective judgments of one form or another will always be required in both risk assessment and risk management decision-making.
We control subjectivity in risk management by:
- Raising awareness of where/when subjective judgments of probability occur in risk assessment and risk management
- Identifying heuristics and biases where they occur
- Improving the understanding of probability among the team and individual experts
- Calibrating experts individually
- Applying knowledge from formal expert elicitation
- Use expert group facilitation when group probability judgments are sought
Each one of these is it’s own, future, post.
Risk Management is about reducing uncertainty
Risk Management is all about eliminating surprise. So to truly start to understand our risks, we need to understand uncertainty, we need to understand the unknowns. Borrowing from Andreas Schamanek’s Taxonomies of the unknown, let’s explore a few of the various taxonomies of what is not known.
Ignorance Map
I’m pretty sure Ann Kerwin first gave us the “known unknowns” and the “unknown knowns” that people still find a source of amusement about former defense secretary Rumsfield.
| Known | Unknown | |
| Known | Known knowns | Known unknowns (conscious ignorance) |
| Unknown | Unknown knowns (tacit knowledge) | Unknown unknowns (meta-ignorance) |
Understanding uncertainty involves knowledge management, this is why a rigorous knowledge management program is a prerequisite for an effective quality management system.
Risk management is then a way of teasing out the unknowns and allowing us to take action:
- Risk assessments mostly easily focus on the ignorance that we are aware of, the ‘known unknowns’.
- Risk assessments can also serve as a tool of teasing out the ‘unknown knowns’. This is why participation of subject matter experts is so critical. Through the formal methodology of the risk assessment we expose and explore tacit knowledge.
- The third kind of ignorance is what we do now know we do not know, the ‘unknown unknowns’. We generally become aware of unknown unknowns in two ways: hindsight (deviations) and by purposefully expanding our horizons. This expansion includes diversity and also good experimentation. It is the hardest, but perhaps, most valuable part of risk management.
Taxonomy of Ignorance
Smithson distinguishes between passive and active ignorance. Passive ignorance involves areas that we are ignorant of, whereas active ignorance refers to areas we ignore. He uses the term ‘error’ for the unknowns encompassed by passive ignorance and ‘irrelevance’ for active ignorance.
Taboo is fascinating because it gets to the heart of our cultural blindness, those parts of our organization that are closed to scrutiny.
Smithson can help us understand why risk assessments are both a qualitative and a quantitative endeavor. While dealing with the unknown is the bread and butter of statistics, only a small part of the terrain of uncertainty is covered. Under Smithson’s typology, statistics primarily operates in the area of incompleteness, across probability and some kinds of vagueness. In terms of its considerations of sampling bias, statistics also has some overlap with inaccuracy. But, as the typology shows, there is much more to unknowns than the areas statistics deals with. This is another reason that subject matter experts, and different ways of thinking is a must.
Ensuring wide and appropriate expert participation gives additional perspectives on unknowns. There is also synergies by finding unrecognized similarities between disciplines and stakeholders in the unknowns they deal with and there may be great benefit from combining forces. It is important to use these concerns to enrich thinking about unknowns, rather than ruling them out as irrelevant.
Sources of Surprise
Risk management is all about managing surprise. It helps to break surprise down to three types: risk, uncertainty and ignorance.
- Risk: The condition in which the event, process, or outcomes and the probability that each will occur is known.
- Issue: In reality, complete knowledge of probabilities and range of potential outcomes or consequences is not usually known and is sometimes unknowable.
- Uncertainty: The condition in which the event, process, or outcome is known (factually or hypothetically) but the probabilities that it will occur are not known.
- Issue: The probabilities assigned, if any, are subjective, and ways to establish reliability for different subjective probability estimates are debatable.
- Ignorance: The condition in which the event, process, or outcome is not known or expected.
- Issue: How can we anticipate the unknown, improve the chances of anticipating, and, therefore, improve the chances of reducing vulnerability?
Effective use of the methodology moves ideally from ignorance to eventually risk.
| Ignorance | Description | Methods of Mitigation |
| Closed Ignorance | Information is available but SMEs are unwilling or unable to consider that some outcomes are unknown to them. | Self-audit process, regular third-party audits, and open and transparent system with global participation |
| Open Ignorance | Information is available and SMEs are willing to recognize and consider that some outcomes are unknown. | |
| Personal | Surprise occurs because an individual SME lacks knowledge or awareness of the available information. | effective teams xxplore multiple perspectives by including a diverse set of individuals and data sources for data gathering and analysis. Transparency in process. |
| Communal | Surprise occurs because a group of SMEs has only similar viewpoints represented or may be less willing to consider views outside the community. | Diversity of viewpoints and sue of tools to overcome group-think and “tribal” knowledge |
| Novelty | Surprise occurs because the SMEs are unable to anticipate and prepare for external shocks or internal changes in preferences, technologies, and institutions. | Simulating impacts and gaming alternative outcomes of various potentials under different conditions (Blue Team/Read Team exercises) |
| Complexity | Surprise occurs when inadequate forecasting tools are used to analyze the available data, resulting in inter-relationships, hidden dependencies, feedback loops, and other negative factors that lead to inadequate or incomplete understanding of the data. | System Thinking Track changes and interrelationships of various systems to discover potential macro-effect force changes 12-Levers |
Risk Management is all about understanding surprise and working to reduce uncertainty and ignorance in order to reduce, eliminate and sometimes accept. As a methodology it is effective at avoiding surrender and denial. With innovation we can even contemplate exploitation. As organizations mature, it is important to understand these concepts and utilize them.
References
- Gigerenzer, Gerd and Garcia-Retamero, Rocio. Cassandra’s Regret: The Psychology of Not Wanting to Know (March 2017), Psychological Review, 2017, Vol. 124, No. 2, 179–196.
- House, Robert J., Paul J. Hanges, Mansour Javidan, Peter Dorfman, and Vipin Gupta, eds. 2004. Culture, Leadership, and Organizations: The GLOBE Study of 62 Societies. Thousand Oaks, Calif.: Sage Publications.
- Kerwin, A. (1993). None Too Solid: Medical Ignorance. Knowledge, 15(2), 166–185.
- Smithson, M. (1989) Ignorance and Uncertainty: Emerging Paradigms, New York: Springer-Verlag.
- Smithson, M. (1993) “Ignorance and Science”, Knowledge: Creation, Diffusion, Utilization, 15(2) December: 133-156.
Knowledge management & effective change management
At ASQ BOSCON 2019 I spoke on “Knowledge management & effective change management.” From my proposal:
An effective change management system includes active knowledge management, leveraging existing process and product knowledge; capturing new knowledge gained during implementation of the change; and, transferring that knowledge in appropriate ways to all stakeholders. This session will focus on three key areas of knowledge management as it enables change. It will provide an understanding of the principles of knowledge management, including: transforming data into information; the acquisition and creation of knowledge; and the some shared best practices for dissemination and using information and knowledge for the purpose of change management and building a quality culture.
Making Learning a Part of Everyday Work
Cultivating expertise, in short learning, is critical to building a quality culture. Yet, the urgency of work easily trumps learning. It can be difficult to carve out time for learning in the inexorable flow of daily tasks. We are all experienced with the way learning ends up being in the lowest box on the 2×2 Eisenhower matrix, or however you like to prioritize your tasks.
For learning to really happen, it must fit around and align itself to our working days. We need to build our systems so that learning is an inevitable result of doing work. There are also things we as individuals can practice to make learning happen.
What we as individuals can do
Practice mindfulness. As you go about your daily job be present and aware, using it as an opportunity to ability to learn and develop. Don’t just sit in on that audit; notice and learn the auditor’s tactics and techniques as you engage with her. Ask product managers about product features; ask experts about industry trends; ask peers for feedback on your presentation skills. These kinds of inquiries are learning experiences and most peers love to tell you what they know.
Keep a to-learn list. Keep a list of concepts, thoughts, practices, and vocabulary you want to explore and then later later explore them when you have a few moments to reflect. Try to work a few off the list, maybe during your commute or at other times when you have space to reflect.
Build learning into your calendar. Many of us schedule email time, time for project updates, time to do administrative work. Make sure you dedicate time for learning.
Share meaningfully. Share with others, but just don’t spread links. Discuss why you are sharing it, what you learned and why you think it is important. This blog is a good example of that.
What we can build into our systems
Make sure our learning and knowledge management systems are built into everything we do. Make them easy to use. Ensure content is shared internally and leads to continuous improvement.
Ensure learning is valued.
Plan for short-term wins. There is no nirvana, no perfect state. Ensure you have lots of little victories and shareable moments. Plan for this as part of your schedules and cycles.
Learning is a very effective lever for system improvement. At the very least it gives us the power to “add, change, evolve or self-organize system structure” (lever 4) and can also start giving us ways to change the paradigm (lever 2) and eventually even transcend paradigms (lever 1).
Investigations of a Dog 