Build Key Risk Indicators

We perform risk assessments; execute risk mitigations; and we end up with four types of inherent risks (parenthesis is opportunities) in our risk register:

Mitigated (or enhanced)
Avoided (or exploited)
Transferred (or shared)
Accepted

We’ve built a set of risk response plans to ensure we are continuing to treat these risks. And now we need to monitor the effectiveness of our risk plan and to ensure that the risks are behaving in the manner anticipated during risk treatment.

The living risk assessment is designed to conduct reassessment of risks after treatment and continuously throughout the life cycle. However, not all systems and risks need to be reassessed continually, and the organization should prioritize which systems should be reassessed based on a schedule.

Identify indicators that inform the organization about the status of the risk without having to conduct a full risk assessment every time. The trending status of these indicators can act as a flag for investigations, which may result in complete risk assessments.

This risk indicator is then a metric that indicates the state of the level of risk. It is important to note that not all indicators show the exact level of risk exposure, instead providing a trend of drivers, causes or intermediary effects of risk.

The most important risks can be categorized as key risks and the indicators for these key risks are known as key risk indicators (KRIs) which can be defined as: A metric that provides a leading or lagging indicator of the current state of risk exposure on key objectives. KRIs can be used to continually assess current and predict potential risk exposures.

These KRIs need to have a strong relationship with the key performance indicators of the organization.

KRIs are monitored through Quality Management Review.

A good rule of thumb is as you identify the key performance indicators to assess the performance of a specific process, product, system or function you then identify the risks and the KRIs for that objective.

Strive to have leading indicators that measure the elements that influences the risk performance. Lagging indicators will measure they actual performance of the risk controls.

These KRIs qualitatively or quantitatively present the risk exposure by having a strong relationship qirh the risk, its intermediate output or its drivers.

Let’s think in terms of a pharmaceutical supply chain. We’ve done our risk assessments and end up with a top level view like this:

For the risk column we should have some good probabilities and impacts and mitigations in place. We can then chose some KRIs to monitor, such as

Nonconformance rate
Supplier score card
Lab error rate
Product Complaints

As we develop, our KRIs can get more specific and focused. A good KRI is:

Quantifiable
Measurable (accurately and precisely)
Can be validated (have a high level of confidence)
Relevant (measuring the right thing associated with decisions)

In developing a KRI to serve as a leading indicator for potential future occurrences of a risk, it can be helpful to think through the chain of events that led to the event so that management can uncover the ultimate driver (i.e., root cause(s)) of the risk event. When KRIs for root cause events and intermediate events are monitored, we are in an enviable position to identify early mitigation strategies that can begin to reduce or eliminate the impact associated with an emerging risk event.

These KRIs will help us monitor and quantify our risk exposure. They help our organizations compare business objectives and strategy to actual performance to isolate changes, measure the effectiveness of processes or projects, and demonstrate changes in the frequency or impact of a specific risk event.

Effective KRIs can provide value to the organization in a variety of ways. Potential value may be derived from each of the following contributions:

Risk Appetite – KRIs require the determination of appropriate thresholds for action at different levels within the organization. By mapping KRI measures to identified risk appetite and tolerance levels, KRIs can be a useful tool for better articulating the risk appetite that best represents the organizational mindset.
Risk and Opportunity Identification – KRIs can be designed to alert management to trends that may adversely affect the achievement of organizational objectives or may indicate the presence of new opportunities.
Risk Treatment – KRIs can initiate action to mitigate developing risks by serving as triggering mechanisms. KRIs can serve as controls by defining limits to certain actions.

The Risk Register

Every organization should ask themselves seven questions about the health of their risk management program.

Do you have a risk management plan?
Have you identified and captured your risks in a risk register?
How have you evaluated and prioritized your risks?
Have you engaged the appropriate stakeholders in the risk identification and evaluation processes?
What about risk owners? Does each risk have a risk owner?
Have the risk owners developed risk response plans for the highest risks?
Are you facilitating a review of your risks periodically, resulting in updates to the risk register and effective risk responses?

At the heart of this program sits the Risk Register, which brings together information about risks to inform those exposed to risks and those who have responsibility for their management. A risk register is used to record and track information about individual risks and how they are being controlled. It can be used to communicate information about risks to stakeholders and highlight particularly important risks. While it can be used at any level of the organization where there are a large number of risks, controls and treatments that need to be tracked, a risk register really shines as a central component of a quality management review. The risk register includes:

List of risks, failure modes or hazards and expected outcomes
A statement about the probability of consequences occurring
Sources or causes of the risk
Priority or risk levels
What is currently being done to control the risk
Risk owner
Actual outcome, if and when available

Risks are generally listed individually as separate events but interdependencies should be flagged.

In recording information about risks, the distinction between risks (the potential effects of what might happen) and risk sources (how or why it might happen) and controls that might fail should be explicit. It can also be useful to indicate the early warning signs that an event might be about to occur.

Many risk registers also include some rating of the significance of a risk, an indication of whether a risk is considered to be acceptable or tolerable, or whether further treatment is needed and the reasons for this decision. Where a significance rating is applied to a risk based on consequences and their likelihood, this should take account of the possibility that controls will fail. A level of risk should not be allocated for the failure of a control as if it were an independent risk.

A risk register is used as the basis for tracking implementation of proposed treatments, so it should contain information about treatments and how they will be implemented, or make reference to other documents or data bases with this information. (Such information can include risk owners, actions, action owners, action business case summaries, budgets and timelines, etc.). This living document can usually roll (or even serve as) the Quality Plan.

Strengths of risk registers include the following.

Information about risks is brought together in a form where actions required can be identified and tracked.
Information about different risks is presented in a comparable format, which can be used to indicate priorities and is relatively easy to interrogate.
The construction of a risk register usually involves many people and raises general awareness of the need to manage risk.

By doing this, the risk register serves as a central underpining for the organization as it builds a risk culture, driving transparency and accountability.

Building Risk Based Thinking in the Organization requires a strong governance structure

Pay attention the the following limitations:

Risks captured in risk registers are typically based on events, which can make it difficult to accurately characterize some forms of risk
The apparent ease of use can give misplaced confidence in the information because it can be difficult to describe risks consistently and sources of risk, risks, and weaknesses in controls for risk are often confused.
There are many different ways to describe a risk and any priority allocated will depend on the way the risk is described and the level of disaggregation of the issue.
Considerable effort is required to keep a risk register up to date (for example, all proposed treatments should be listed as current controls once they are implemented, new risks should be continually added and those that no longer exist removed).
Risks are typically captured in risk registers individually. This can make it difficult to consolidate information to develop an overall treatment program.

Artifacts, like the risk register, both demonstrate and channel culture. Invest the time in your organization’s register, and you will reap dividends towards developing a risk friendly culture.

Evolved Expendable Launch Vehicle (EELV) Quality Management

We determined that ULA, SpaceX, and AR were not performing adequate quality assurance management for the EELV program as evidenced by the 181 nonconformities to the AS9100C at the EELV contractor production facilities. This inadequate quality assurance management could increase costs, delay launch schedules, and increase the risk of mission failure.

From ”

Evaluation of the Evolved Expendable Launch Vehicle Program Quality Management System DODIG-2018-045, Department of Defense Office of Inspector General

It is useful to read audit reports and inspection findings from multiple industries. From this we can see trends, make connections and learn.

I see a few things that stand out.

DOD findings

Risk Register

Our evaluation of the RIO database showed that 11 out of 26 risks related to either Atlas V or Delta IV launch vehicle were in “red” status, which indicates that risk mitigation was behind schedule.

It is not enough to identify risks (though that is a critical place to start). You just can’t track them (though again, if you don’t track it you don’t see it). You actually have to have clear plans to mitigate and eliminate the risks. And this is where the program seemed to fall short.

Not a surprise. I think a lot of companies are having these difficulties. In the pharma world the regulatory agencies have been signaling pretty strongly that this is an issue.

Make sure you identify risks, track them, and have plans that are actually carried out to remediate.

Configuration Management

SpaceX failed to comply with AS9100C, section 7.1.3, which requires it to “establish, implement, and maintain a configuration management process.” Configuration management is a controlled process to establish the baseline configuration of a product and any changes to that product. This process should occur during the entire life cycle of a product to provide visibility and control of its physical, functional, and performance attributes.

First rule of reading inspection reports: Things probably went bad if a section starts with standard review 101 material.

That said, hello change management my dear friend.

This was the gist of my ASQ WCQI workshop last May, every industry needs good change management and change control.

Material Management

ULA and AR failed to comply with AS9100C, section 8.3, which requires them to “ensure that product which does not conform to product requirements are identified and controlled to prevent its unintended use or delivery.”

At ULA, we found 18 expired limited-life material items that were between 32 and 992 days past their expiration dates, but available for use on EELV flight hardware. This material should have been impounded and dispositioned. The use of expired limited-life items, such as glues and bonding agents, could result in product that does not meet specifications and may require costly rework.

I find it hard to believe that these companies aren’t tracking inventory. If they are tracking inventory and have any sort of cycle count process then the mechanism exists to ensure expired material is removed from the possibility of use. And yet we still see these observations across the pharma industry as well.

Concluding Thoughts

Quality Management has it its core the same principles, no matter the industry. We use similar tools. Leverage the best practices out there. Read about stresses other companies are having, learn from them and remediate at your own organization.