Probing Unknown Unknowns

In the post “Risk Management is about reducing uncertainty,” I discussed ignorance and surprise, covering the idea of “unknown unknowns”, those things that we don’t even know that we don’t know.

Our goal should always be to reduce ignorance. Many unknown unknowns are just things no one has bothered to find out. What we need to do is ensure our processes and systems are constructed so that they recognize unknowns.

There are six factors that need to be explored to find the unknown unknowns.

  1. Complexity: A complex process/system/project contains many interacting elements that increase the variety of its possible behaviors and results. Complexity increases with the number, variety, and lack of robustness of the elements of the process, system or project.
  2. Complicatedness: A complicated process/system/project involves many points of failure, the ease of finding necessary elements and identifying cause-and-effect relationships; and the experts/participants aptitudes and experiences.
  3. Dynamism: The volatility or the propensity of elements and relationships to change.
  4. Equivocality: Knowledge management is a critical enabler of product and project life cycle management. If the information is not crisp and specific, then the people who receive it will be equivocal and won’t be able to make firm decisions. Although imprecise information itself can be a known unknown, equivocality increases both complexity and complicatedness. 
  5. Perceptive barriers: Mindlessness. This factor includes a lot of our biases, including an over-reliance on past experiences and traditions, the inability to detect weak signals and ignoring input that is inconvenient or unappealing.
  6. Organizational pathologies: Organizations have problems, culture can have weaknesses. These structural weaknesses allow unknown unknowns to remain hidden.
Interrogating Knowable Unknown Unknowns

The way to address these six factors is to evaluate and challenge by using the following approaches:

Interviewing

Interviews with stakeholders, subject matter experts and other participants can be effective tools for uncovering lurking problems and issues. Interviewers need to be careful not to be too enthusiastic about the projects they’re examining and not asking “yes or no” questions. The best interviews probe deep and wide.

Build Knowledge by Decomposing the System/Process/Project

Standard root cause analysis tools apply here, break it down and interrogate all the subs.

  1. Identifying the goals, context, activities and cause-effect relationships
  2. Breaking the domains into smaller elements — such as processes, tasks and stakeholders
  3. Examining the complexity and uncertainty of each element to identify the major risks (known unknowns) that needed managing and the knowledge gaps that pointed to areas of potential unknown unknowns.

Analyze Scenarios

Construct several different future outlooks and test them out (mock exercises are great). This approach accepts uncertainty, tries to understand it and builds it into the your knowledge base and reasoning. Rather than being predictions, scenarios are coherent and credible alternative futures built on dynamic events and conditions that are subject to change.

Communicate Frequently and Effectively

Regularly and systematically reviewing decision-making and communication processes, including the assumptions that are factored into the processes, and seeking to remove information asymmetries, can help to anticipate and uncover known unknowns. Management Review is part of this, but not the only component. Effective and frequent communication is essential for adaptability and agility. However, this doesn’t necessarily mean communicating large volumes of information, which can cause information overload. Rather, the key is knowing how to reach the right people at the right times. Some important aspects include:

  • Candor: Timely and honest communication of missteps, anomalies and missing competencies. Offer incentives for candor to show people that there are advantages to owning up to errors or mistakes in time for management to take action. It is imperative to eliminate any perverse incentives that induce people to ignore emerging risks.
  • Cultivate an Alert Culture: A core part of a quality culture should be an alert culture made up of people who strive to illuminate rather than hide potential problems. Alertness is built by: 1) emphasizing systems thinking; 2) seek to include and build a wide range of experiential expertise — intuitions, subtle understandings and finely honed reflexes gained through years of intimate interaction with a particular natural, social or technological system; and 3) learn from surprising outcomes.

By working to evaluate and challenge, to truly understand our systems and processes, our risk management activities will be more effective and truly serve to make our systems resilient.

Recommended Reading

Interviewing

One of the great tools of root cause analysis, planning, process improvement and knowledge management is the interview. Properly used the interview allows one to gather a great deal of information and perspective and ferret out hidden information.

For interviews to be truly effective, we have to understand how the function and apply a process. Cognitive Interviewing, originally created for law enforcement and later adopted during accident investigations by the National Transportation Safety Board (NTSB), provides an effective framework. I was first introduced to this at my previous company, where it has become a real linchpin, so I share it here.

The two principles here are:

  • Witnesses need time and encouragement to recall information
  • Retrieval cues enhance memory recall

Based on these two principles there are four components:

ComponentWhat It Consists of
Mental ReinstatementEncourage the interviewee to mentally recreate the environment and people involved.
In-Depth ReportingEncourage the reporting of all the details, even if it is minor or not directly related to the purpose of the interview. This is intended to improve the detail and accuracy of memory.

For example, if investigation a computer error, you would encourage the interviewee to discuss everything they were doing around the event. You would hold the interview at the station the error happened, ideally using the computeras much as possible.
Multiple PerspectivesAsk the interviewee to recall the event from others’ points of view. For example, the person upstream or downstream, or a partner or observer.
Several OrdersAsk the interviewee to recount the timeline in different ways. Beginning to end, end to beginning.
Four Components of Cognitive Interviewing

A key part of this is that retrieval cues access memory. This is why doing the interview on the scene (or Gemba) is so effective.

The basic behaviors you want to bring to bear are:

  • Recreate the original context; have them outline and walk you through process to explain how they work.
  • Tell the the witness to actively generate information and not wait passively for the interviewer to ask questions.
  • Adopt the witness’s perspective; ask eyewitness-compatible questions.
  • Perform the interview at the Gemba, the place where the work happens.
  • Listen actively, do not interrupt, and pause after the witness’s response.
  • Ask open-ended questions, utilize short phrases when possible.
  • Encourage the witness to use imagery. Explicitly request detailed descriptions.
  • Follow the sequence of the cognitive interview major components.
  • Bring support materials such as attachments, procedures, and copies of relevant documents.
  • Establish a connection with the witness; demeanor has a big impact.
  • Remember, active listening.
  • Do not tell the interviewee how they made the mistake, blame, or assume.

Active listening is key here.

Active Listening funnel

At the mouth of the funnel we begin with an ‘open’ question. This question is intended to give the interviewee the widest possible scope for responding. Sometimes it may be necessary to repeat or rephrase this question to give the interviewee more thinking time and further opportunities to raise information. Working down the narrowing body of the funnel we use a series of probing questions to draw out further specific information and help complete the picture. Closed questions then have their place to draw out, check or confirm specific pieces of information, or to get the interviewee to commit on a point more precisely. This then brings us to the bottom of the funnel where we clarify, using a short summary, what we have got out of the discussion, aiming to check our understanding of the main points. The question sequence might go something like this:

  • ‘Tell me how you went about…?’ (open)
  • ‘How did you prepare?’ (open – secondary)
  • ‘What was your starting point?’ (probe)
  • ‘So, what happened next?’ (probe)
  • ‘Who else was involved?’ (probe)
  • ‘And how did they respond?’ (probe)
  • ‘What were your thoughts at that stage?’ (probe)
  • ‘What were the main outcomes?’ (probe)
  • ‘So, that took a total of 30 minutes?’ (closed – clarifying)
  • ‘And the task was completed?’ (closed – clarifying)
  • ‘So, let me see if I’ve followed you…’ (checking – summary)

A good interview requires preparation. Have opening questions ready, ensure you have all the right props and the right people involved. That extra hour or two will pay dividends.

Here is a helpful worksheet.

Bias

There are many forms of bias that we must be cognizant during problem solving and decision making.

That chart can be a little daunting. I’m just going to mention three of the more common biases.

  • Attribution bias: When we do something well, we tend to think it’s because of our own merit. When we do something poorly, we tend to believe it was due to external factors (e.g. other people’s actions). When it comes to other people, we tend to think the opposite – if they did something well, we consider them lucky, and if they did something poorly, we tend to think it’s due to their personality or lack of skills.
  • Confirmation bias: The tendency to seek out evidence that supports decisions and positions we’ve already embraced – regardless of whether the information is true – and putting less weight on facts that contradict them.
  • Hindsight bias: The tendency to believe an event was predictable or preventable when looking at the sequence of events in hindsight. This can result in oversimplification of cause and effect and an exaggerated view that a person involved with an event could’ve prevented it. They didn’t know the outcome like you do now and likely couldn’t have predicted it with the information available at the time.

A few ways to address our biases include:

  • Bouncing ideas off of others, especially those not involved in the discussion or decision.
  • Surround yourself with a diverse group of people and do not be afraid to consider dissenting views. Actively listen.
  • Imagine yourself in other’s shoes.
  • Be mindful of your internal environment. If you’re struggling with a decision, take a moment to breathe. Don’t make decisions tired, hungry or stressed.
  • Consider who is impacted by your decision (or lack of decision). Sometimes, looking at how others will be impacted by a given decision will help to clarify the decision for you.

The advantage of focusing on decision quality is that we have a process that allows us to ensure we are doing the right things consistently. By building mindfulness we can strive for good decisions, reducing subjectivity and effective problem-solving.