Effectiveness I recently had a bit of a wake-up call via Twitter. I asked the following question: “What’s the one thing /above all/ that makes for an effective organisation?” My thanks to all those who took the time to reply with their viewpoint. The wake-up call for me was the variety of these responses. All […]
In quality management systems, it is critical to look at effectiveness. If you do not measure, you do not know if the system is working the ways you expect and desire.
We often discuss lagging (output measurement) and leading (predictive) indicators, and this is a good way to start, but if we apply System Thinking and use Meadow’s twelve leverage points we can see that most metrics tend to be around 7-12, with the more effective levers being the least utilized.
I think there are a lot of value in finding metrics within these levers.
So for example, a few indicators on the effectiveness of lever 4 “The Power to Add, Change, Evolve, or Self-Organize System Structure”:
Lagging
Leading
Effective CAPAs to the System
Number of changes initiated by level of organization and scale of change
Risk Management is a key enabler of any quality by design, whether of product, facility/equipment or study. We do living risk assessments to understand the scope of our ongoing risk. Inevitably we either want to implement that new or improved design or we want to mitigate the ongoing risks in our operation. So we turn to change management. And as part of that change management we do a risk assessment. Our change management then informs ongoing risk review.
Risk Management Leads to Change Management
Design Implementation
Through the iterative design lifecycle there is a final design ready for introduction. Perhaps this is a totally new thing, perhaps it is a new set of equipment or processes, or just a modification.
All along through the iterative design lifecycle risk management has been applied to establish measurable, testable, unambiguous and traceable performance requirements. Now your process engages with change management to introduce the change.
And a new risk assessment is conducted.
This risk assessment is asking a different question. During the iterative design lifecycle the risk question is some form of “What are the risks from this design on the patient/process.” As part of risk management, the question is “What are the risks to SISPQ/GMP from introducing the change.”
This risk assessment is narrower, in that it looks at the process of implementing. Broader that it looks at the entirety of your operations: facility, supply chain, quality system, etc.
The design risk assessment and risk management activities informs the change management risk assessment, but it cannot replace them. They also can serve to lower the rigor of the change management risk assessment, allowing the use of a less formal tool.
Living Risk Reviews
In the third phase of risk management – risk review – we confirm that the risks identified and mitigated as planned and are functioning as intended. We also evaluate to see if any additional, previously unpredicted risks have appeared. Risk review is the living part of the lifecycle as we return to it on a periodic basis.
From this will come new mitigations, targeted to address the identified risks. These mitigations inevitably lead to change management.
We again do a new risk assessment focusing on the risk of implementing the change. Informed by the living risk assessment, we can often utilize a less formal tool to look at the full ramifications of introducing the mitigation (a change).
Each and every change requires a risk assessment to capture the risks of the change. This ICHQ10 requirement is the best way to determine if the change is acceptable.
This risk assessment evaluates the impact on the change on the facility, equipment, materials, supply chain, processes. testing, quality systems and everything else. It is one of the critical reasons it is crucial to involve the right experts.
From this risk assessment comes the appropriate actions before implementing the change, as well as appropriate follow-up activities and it can help define the effectiveness review.
Depends. Sometimes the risk management looks at the individual implementations. Othertimes you need to do separate ones. Many times the risk assessment lead you to breaking up one change control into many. Evaluate as follows:
Are the risks from the separate implementations appropriately captured
Are the risks from pauses between implementations appropriately captured
As the ripples appropriately understood
Change Management Leads back to Risk Management
Sometimes a change control requires a specific risk assessment to be updated, or requires specific risk management to happen.
What about HAACP?
Hazard Analysis Critical Control Point (HACCP) are great tools for risk assessments. They are often the catalyst for doing a change, they are often the artifact of a change. They should never be utilized for determining the impact of a change.
A hazard is any biological, chemical, or physical property that impacts human safety. The HAACP identifies and establishes critical limits. But a HAACP is not the tool to use to determine if a change should move forward and what actions to do. It is to static.
In Closing
Risk Management is an enabler for change, a tenet enshrined in the ICH guidances. We are engaging in risk management activities throughout our organizations. It is critical to understand how the various risk management activities fit together and how they should be separated.
There is little argument regarding the critical role that structured problem-solving plays in a lean transformation. Besides the business results associated with solving problems, developing problem-solving skills increases learning, drives the desired change in thinking, and helps people more clearly understand how lean works as a system. With this said, however, it is amazing how little effort many organizations put into developing effective problem-solving skills. It seems like more time is spent on things like 5S, value stream mapping, and other tools that are generally considered easier to apply and less likely to be met with resistance. As a result, transformation does not occur, improvements are not sustainable, and the big gains possible through lean thinking are never achieved.
Good discussion on the importance of rigorous, sustained problem-solving as part of Lean initiatives. I think many of us have experienced this in our own organizations.
Utilizing problem solving tools in a structured way helps us better understand what is happening, how it is happening and most importantly, why it is happening. Armed with this understanding we can then engage in those improvements. Problem solving is key to getting those improvements because it allows us to discover why a problem is actually happening and not to just treat symptoms.
Problem Solving needs to reach a level of detail that accurately identifies an actionable cause that can then be addressed.
We determined that ULA, SpaceX, and AR were not performing adequate quality assurance management for the EELV program as evidenced by the 181 nonconformities to the AS9100C at the EELV contractor production facilities. This inadequate quality assurance management could increase costs, delay launch schedules, and increase the risk of mission failure.
It is useful to read audit reports and inspection findings from multiple industries. From this we can see trends, make connections and learn.
I see a few things that stand out.
Risk Register
Our evaluation of the RIO database showed that 11 out of 26 risks related to either Atlas V or Delta IV launch vehicle were in “red” status, which indicates that risk mitigation was behind schedule.
It is not enough to identify risks (though that is a critical place to start). You just can’t track them (though again, if you don’t track it you don’t see it). You actually have to have clear plans to mitigate and eliminate the risks. And this is where the program seemed to fall short.
Not a surprise. I think a lot of companies are having these difficulties. In the pharma world the regulatory agencies have been signaling pretty strongly that this is an issue.
Make sure you identify risks, track them, and have plans that are actually carried out to remediate.
Configuration Management
SpaceX failed to comply with AS9100C, section 7.1.3, which requires it to “establish, implement, and maintain a configuration management process.” Configuration management is a controlled process to establish the baseline configuration of a product and any changes to that product. This process should occur during the entire life cycle of a product to provide visibility and control of its physical, functional, and performance attributes.
First rule of reading inspection reports: Things probably went bad if a section starts with standard review 101 material.
This was the gist of my ASQ WCQI workshop last May, every industry needs good change management and change control.
Material Management
ULA and AR failed to comply with AS9100C, section 8.3, which requires them to “ensure that product which does not conform to product requirements are identified and controlled to prevent its unintended use or delivery.”
At ULA, we found 18 expired limited-life material items that were between 32 and 992 days past their expiration dates, but available for use on EELV flight hardware. This material should have been impounded and dispositioned. The use of expired limited-life items, such as glues and bonding agents, could result in product that does not meet specifications and may require costly rework.
I find it hard to believe that these companies aren’t tracking inventory. If they are tracking inventory and have any sort of cycle count process then the mechanism exists to ensure expired material is removed from the possibility of use. And yet we still see these observations across the pharma industry as well.
Concluding Thoughts
Quality Management has it its core the same principles, no matter the industry. We use similar tools. Leverage the best practices out there. Read about stresses other companies are having, learn from them and remediate at your own organization.
Investigations of a Dog 