Likelihood of occurrence in risk estimation

People use imprecise words to describe the chance of events all the time — “It’s likely to rain,” or “There’s a real possibility they’ll launch before us,” or “It’s doubtful the nurses will strike.” Not only are such probabilistic terms subjective, but they also can have widely different interpretations. One person’s “pretty likely” is another’s “far from certain.” Our research shows just how broad these gaps in understanding can be and the types of problems that can flow from these differences in interpretation.

“If You Say Something Is “Likely,” How Likely Do People Think It Is?” by by Andrew Mauboussin and Michael J. Mauboussin

Risk estimation is based on two components:

The probability of the occurrence of harm
The consequences of that harm

With a third element of detectability of the harm being used in many tools.

Often-times we simplify probability of the occurrence into likelihood. The quoted article above is a good simple primer on why we should be careful of that. It offers three recommendations that I want to talk about. Go read the article and then come back.

I. Use probabilities instead of words to avoid misinterpretation

Avoid the simplified quality probability levels, such as “likely to happen”, “frequent”, “can happen, but not frequently”, “rare”, “remote”, and “unlikely to happen.” Instead determine probability levels. even if you are heavily using expert opinion to drive probabilities, given ranges of numbers such as “<10% of the time”, “20-60% of the time” and “greater than 60% of the time.”

It helps to have several sets of scales.

The article has an awesome graph that really is telling for why we should avoid words.

W180614_MAUBOUSSIN_HOWPEOPLE

II. Use structured approaches to set probabilities

Ideally pressure test these using a Delphi approach, or something similar like paired comparisons or absolute probability judgments. Using the historic data, and expert opinion, spend the time to make sure your probabilities actually capture the realities.

Be aware that when using historical data that if there is a very low frequent of occurrence historically, then any estimate of probability will be uncertain. In these cases its important to use predicative techniques and simulations. Monte Carlo anyone?

III. Seek feedback to improve your forecasting

Risk management is a lifecycle approach, and you need to be applying good knowledge management to that lifecycle. Have a mechanism to learn from the risk assessments you conduct, and feed that back into your scales. These scales should never be a once and done.

In Conclusion

Risk Management is not new. It’s been around long enough that many companies have the elements in place. What we need to be doing to driving to consistency. Drive out the vague and build best practices that will give the best results. When it comes to likelihood there is a wide body of research on the subject and we should be drawing from it as we work to improve our risk management.

Move beyond setting your scales at the beginning of a risk assessment. Scales should exist as a library (living) that are drawn upon for specific risk evaluations. This will help to ensure that all participants in the risk assessment have a working vocabulary of the criteria, and will keep us honest and prevent any intentional or unintentional manipulation of the criteria based on an expected outcome.

Training assessment as part of change management

One of the key parts of any change (process improvement, project, etc) is preparing people to actually do the work effectively. Every change needs to train.

Building valid and reliable training at the right level for the change is critical. Training is valid when it is tied to the requirements of the job – the objectives; and when it includes evaluations that are linked to the skills and knowledge started in the objectives. Reliability means that the training clearly differentiates between those who can perform the task and those who cannot.

A lot of changes default to read-and-understand training. This quite bluntly is the bane of valid and reliable training with about zero value and would be removed from our toolkit if I had my way.

There are a lot of training models, but I hold there is no single or best method. The most effective and efficient combination of methods should be chosen depending on the training material to be covered and the specific needs of the target group.

For my purposes I’ll draw from Edgar Dale’s Cone of Experience, which incorporates several theories related to instructional design and learning processes. Dale theorized that how a learner retained information is based on what they “do” as opposed to what is “heard,” “read” or “observed.” This is often called experiential or action learning.

dalescone

Based on this understanding we can break the training types down. For example:

Structured discussions are Verbal and some Visual, and lives within the Abstract
Computer Based Trainings are mostly Iconic, with a few concrete
Instructor Led Trainings are a lot about Concrete
On-the-job training is all about the Concrete

Once we have our agreed upon training methods and understand what makes them a good training we can then determine what criteria of a change leads to the best outcome for training. Some example criteria include:

Is a change in knowledge or skills needed to execute the procedure?
Is the process or change complex? Are there multiple changes?
Criticality of Process and risk of performance error? What is the difficulty in detecting errors?
What is the identified audience (e.g., location, size, department, single site vs. multiple sites)?
Is the goal to change workers‘ conditioned behavior

This sort of questioning gets us to risk based thinking. We are determining where the biggest bang from our training is.

Building training is a different set of skills. I keep threatening a training peer with doing a podcast episode (probably more than one) on the subject (do I really want to do podcasts?).

The last thing I want to leave you is build training evaluations into this. Kilpatrick’s model is a favorite – Level 4 Results evaluations which tell us how effective our training was overtime actually makes a darn good effectiveness review. I strongly recommend building that into a change management process.

Knowledge Management

ICH Q10 “Pharmaceutical Quality System” describes a lifecycle approach, from development through product discontinuation. The knowledge about a pharmaceutical product and the processes required to reliably produce that product starts with product and process development. An effective pharmaceutical quality system (PQS) uses the knowledge acquired throughout the lifecycle of the product, builds on that knowledge, and applies it to:

Other stages of the product lifecycle
Other product lifecycles

A change management system is defined as an important element of a PQS as seen in this figure reproduced from ICH Q10.

Q10

There are two enablers to this quality system model, knowledge management and risk management. The thing about those enablers is that they are really intertwined. Or put another way, risk management is a powerful way to make use of your knowledge.

ICHQ12 “Technical and Regulatory Considerations for Pharmaceutical Product Lifecycle Management” (in draft) expands on knowledge management and provides more examples of its use. The below illustration is an adaptation of one found in the draft Q12.

knowledge and change

There are many ways to tap into knowledge management in change management. The subject matter experts are critical, as is checklists and risk ranking and filtering tools. Knowledge should drive the development of an effectiveness review.

One of my favorite is the Living Risk Assessment approach. Living risk assessments are a holistic view of a system, product, or process in an effort to prevent risk realization. They are updated throughout the product /system lifecycle to continuously assess risks that may arise or change.

In the context of change management, the living risk assessment is both an input and an output. A rigorous, maintained, living risk assessment allows us to prospectively mitigate potential risks as part of our change management program.

Living Risk Assessments have a schedule, a review period (for example, once a year) to evaluate how risk has changed, drawing from all the sources of knowledge. It is also important to have a way to trigger adhoc reviews (for example, major process changes or critical deviations).

living risk assessments

In my ASQ World Conference workshop I will be going into more detail on knowledge management, risk management and the pharmaceutical quality system. I’ll also be discussing what non-Pharma companies can learn from the PQS.