Lessons Learned and Change Management

One of the hallmarks of a quality culture is learning from our past experiences, to eliminate repeat mistakes and to reproduce success. The more times you do an activity, the more you learn, and the better you get (within limits for simple activities).  Knowledge management is an enabler of quality systems, in part, to focus on learning and thus accelerate learning across the organization as a whole, and not just one person or a team.

This is where the” lessons learned” process comes in.  There are a lot of definitions of lessons learned out there, but the definition I keep returning to is that a lessons learned is a change in personal or organizational behavior as a result from learning from experience. Ideally, this is a permanent, institutionalized change, and this is often where our quality systems can really drive continuous improvement.

Lessons learned is activity to lessons identified to updated processes
Lessons Learned

Part of Knowledge Management

The lessons learned process is an application of knowledge management.

Lessons identified is generate, assess, and share.

Updated processes (and documents) is contextualize, apply and update.

Lessons Learned in the Context of Knowledge Management

Identify Lessons Learned

Identifying lessons needs to be done regularly, the closer to actual change management and control activities the better. The formality of this exercise depends on the scale of the change. There are basically a few major forms:

  • After action reviews: held daily (or other regular cycle) for high intensity learning. Tends to be very focused on questions of the day.
  • Retrospective: Held at specific periods (for example project gates or change control status changes. Tends to have a specific focus on a single project.
  • Consistency discussions: Held periodically among a community of practice, such as quality reviewers or multiple site process owners. This form looks holistically at all changes over a period of time (weekly, monthly, quarterly). Very effective when linked to a set of leading and lagging indicators.
  • Incident and events: Deviations happen. Make sure you learn the lessons and implement solutions.

The chosen formality should be based on the level of change. A healthy organization will be utilizing all of these.

Level of ChangeForm of Lesson Learned
TransactionalConsistency discussion
After action (when things go wrong)
OrganizationalRetrospective
After action (weekly, daily as needed)
TransformationalRetrospective
After action (daily)

Successful lessons learned:

  • Are based on solid performance data: Based on facts and the analysis of facts.
  • Look at positive and negative experiences.
  • Refer back to the change management process, objectives of the change, and other success criteria
  • Separate experience from opinion as much as possible. A lesson arises from actual experience and is an objective reflection on the results.
  • Generate distinct lessons from which others can learn and take action. A good action avoids generalities.

In practice there are a lot of similarities between the techniques to facilitate a good lessons learned and a root cause analysis. Start with a good core of questions, starting with the what:

  • What were some of the key issues?
  • What were the success factors?
  • What worked well?
  • What did not work well?
  • What were the challenges and pitfalls?
  • What would you approach differently if you ever did this again?

From these what questions, we can continue to narrow in on the learnings by asking why and how questions. Ask open questions, and utilize all the techniques of root cause analysis here.

Then once you are at (or close) to a defined issue for the learning (a root cause), ask a future-tense question to make it actionable, such as:

  • What would your advice be for someone doing this in the future?
  • What would you do next time?

Press for specifics. if it is not actionable it is not really a learning.

Update the Process

Learning implies memory, and an organization’s memories usually require procedures, job aids and other tools to be updated and created. In short, lessons should evolve your process. This is often the responsibility of the change management process owner. You need to make sure the lesson actually takes hold.

Differences between effectiveness reviews and lesson’s learned

There are three things to answer in every change

  1. Was the change effective – did it meet the intended purposes
  2. Did the change have any unexpected effects
  3. What can we learn from this change for the next change?

Effectiveness reviews are 1 and 2 (based on a risk based approach) while lessons learned is 3. Lessons learned contributes to the health of the system and drives continuous improvements in the how we make changes.

Citations

  • Lesson learned management model for solving incidents. (2017). 2017 12th Iberian Conference on Information Systems and Technologies (CISTI), Information Systems and Technologies (CISTI), 2017 12th Iberian Conference On, 1.
  • Fowlin, J. j & Cennamo, K. (2017). Approaching Knowledge Management Through the Lens of the Knowledge Life Cycle: a Case Study Investigation. TechTrends: Linking Research & Practice to Improve Learning61(1), 55–64. 
  • Michell, V., & McKenzie, J. (2017). Lessons learned: Structuring knowledge codification and abstraction to provide meaningful information for learning. VINE: The Journal of Information & Knowledge Management Systems47(3), 411–428.
  • Milton, N. J. (2010). The Lessons Learned Handbook : Practical Approaches to Learning From Experience. Burlington: Chandos Publishing.
  • Paul R. Carlile. (2004). Transferring, Translating, and Transforming: An Integrative Framework for Managing Knowledge across Boundaries. Organization Science, (5), 555.
  • Secchi, P. (Ed.) (1999). Proceedings of Alerts and Lessons Learned: An Effective way to prevent failures and problems. Technical Report WPP-167. Noordwijk, The Netherlands: ESTEC

ASQ Lean and Six Sigma Conference – Speaker

I will be presenting at the ASQ Lean and Six Sigma Conference on March 5, 2019 on the topic “Training as Part of Lean Change Management.”

If you plan on being at the conference, let me know. I always enjoy sitting down with colleagues and chatting.

This topic unites three of my passions: change management, knowledge management and continuous improvements. 

One of the key parts of any change stemming from a project is preparing people to actually do the work effectively. Every change needs to train and building valid and reliable training at the right level for the change is critical.

Training is valid when it is tied to the requirements of the job – the objectives; and when it includes evaluations that are linked to the skills and knowledge started in the objectives. Reliability means that the training clearly differentiates between those who can perform the task and those who cannot.

In this session we will take a risk based training approach to the best outcome for training. The following criteria will be examined and a tool provided for decision making:

  • Is a change in knowledge or skills needed to execute the changed process?
  • Is the process or change complex? Are there multiple changes?
  • Criticality of Process and risk of performance error? What is the difficulty in detecting errors?
  • What is the identified audience (e.g., location,size, department, single site vs. multiple sites)?
  • Is the goal to change workers conditioned behavior?

Armed with these criteria, participants will then be exposed to specific training tools to enable quick adoption of the training:reader-doer, pre-job briefings, and structured discussions. Advantages of each method, as well as common mistakes will be evaluated.

Knowledge management as a key enabler to lean improvements will be examined. Participants will gain an understanding of how to draw from their organizations formal and informal knowledge management systems, and gain an understanding a tool to ensure results of a lean project feedback into the knowledge management system.

Participants will leave this training with the ability to execute decision making around providing successful training for their lean projects and ensuring that this deepens their organization’s knowledge and the ability to apply that knowledge in the future.

Change Management – Post Change Evaluation and Action

In “Change Management – Post Change Evaluation and Action” John Hunter (of CuriousCat) writes a nice post on the The W. Edwards Deming Institute Blog on linking change management to the PDSA (PDCA) lifecycle focusing on Act.

Post Change Evaluation is often called the effectiveness review, and is a critical part of change in the pharmaceutical quality system, and frankly is important no matter the industry.

An effectiveness review is the success criteria of the change viewed over enough data points based on a methodology informed by the nature of the change and risk.

 The success criteria should be achieved. If not, reasons why they have not been achieved should be assessed along with the mitigation steps to address the reasons why, including reverting to the previous operating state where appropriate. This may require the proposal of a subsequent change or amendment of the implementation plan to ensure success. Here we see the loop aspects of the PDSA lifecycle.

All changes should have a way back into knowledge management. The knowledge gathered from implementation of the change should be shared with the development function and other locations, as appropriate, to ensure that learning can be applied in products under development or to similar products manufactured at the same or other locations.

When choosing success criteria always strive for leading indicators that tell you how the change is working. Deviations are an awful way to judge the effectiveness of the change. Instead look for walkthroughs, checklists, audits, data gathering. Direct observation and real-time gathering and analysis of data of any sort is the best.

As mentioned above, ensure the change management/change control system is set up to deal with the inevitable change that does not work. Have a clear set of instructions on how to make that decision (returning to the success criteria), what steps to take to mitigate and what to do next. For example having guidance of when to create a deviation and on how to make a decision to rollback versus implement another change.

Risk Management leads to Change Management, Change Management contains Risk Management

We did an FMEA for the design of the room. Why do we need a risk assessment for the change control to implement the design features?

We have an environmental risk management plan, including a HAACP. Why does this change control require a new risk assessment?

If I received a nickel……

I want to expand on my earlier thoughts on risk management enabling change.

Risk Management is a key enabler of any quality by design, whether of product, facility or equipment. We do living risk assessments to understand the scope of our ongoing risk. Inevitably we either want to implement that new or improved design or we want to mitigate the ongoing risks in our operation. So we turn to change management. And as part of that change management we do a risk assessment. Our change management then informs ongoing risk review.

Risk Management Leads to Change Management

Design Implementation

Through your iterative design lifecycle there is a final design ready for introduction. Perhaps this is a totally new thing, perhaps it is a new set of equipment or processes, or just a modification.

All along through the iterative design lifecycle risk management has been applied to establish measurable, testable, unambiguous and traceable performance requirements. Now your process engages with change management to introduce the change.

And a new risk assessment is conducted.

This risk assessment is asking a different question. During the interative design lifecycle the risk question is some form of “What are the risks from this design on the patient/process.” As part of risk management, the question is “What are the risks to SISPQ/GMP from introducing the change.”

This risk assessment is narrower, in that it looks at the process of implementing. Broader that it looks at the entirety of your operations: facility, supply chain, quality system, etc.

The design risk assessment and risk management activities informs the change management risk assessment, but it cannot replace them. They also can serve to lower the rigor of the change management risk assessment, allowing the use of a less formal tool.

Living Risk Reviews

risk leads to change

In the third phase of risk management – risk review – we confirm that the risks identified and mitigated as planned and are functioning as intended. We also evaluate to see if any additional, previously unpredicted risks have appeared. Risk review is the living part of the lifecycle as we return to it on a periodic basis.

From this will come new mitigations, targeted to address the identified risks. These mitigations inevitably lead to change management.

We again do a new risk assessment focusing on the risk of implementing the change. Informed by the living risk assessment, we can often utilize a less formal tool to look at the full ramifications of introducing the mitigation (a change).

Change Controls contains Risk Management

risk and change management connections

Effective change management is enabled by risk management.

Each and every change requires a risk assessment to capture the risks of the change. This ICHQ10 requirement is the best way to determine if the change is acceptable.

This risk assessment evaluates the impact on the change on the facility, equipment, materials, supply chain, processes. testing, quality systems and everything else. It is one of the critical reasons it is crucial to involve the right experts.

From this risk assessment comes the appropriate actions before implementing the change, as well as appropriate follow-up activities and it can help define the effectiveness review.

What about grouped change controls?

Depends. Sometimes the risk management looks at the individual implementations. Othertimes you need to do separate ones. Many times the risk assessment lead you to breaking up one change control into many. Evaluate as follows:

  • Are the risks from the separate implementations appropriately captured
  • Are the risks from pauses between implementations appropriately captured
  • As the ripples appropriately understood

Change Management Leads back to Risk Management

Sometimes a change control requires a specific risk assessment to be updated, or requires specific risk management to happen.

What about HAACP?

Hazard Analysis Critical Control Point (HACCP) are great tools for risk assessments. They are often the catalyst for doing a change, they are often the artifact of a change. They should never be utilized for determining the impact of a change.

A hazard is any biological, chemical, or physical property that impacts human safety. The HAACP identifies and establishes critical limits. But a HAACP is not the tool to use to determine if a change should move forward and what actions to do. It is to static.

In Closing

Risk Management is an enabler for change, a tenet enshrined in the ICH guidances. We are engaging in risk management activities throughout our organizations. It is critical to understand how the various risk management activities fit together and how they should be separated.

Evolved Expendable Launch Vehicle (EELV) Quality Management

We determined that ULA, SpaceX, and AR were not performing adequate quality assurance management for the EELV program as evidenced by the 181 nonconformities to the AS9100C at the EELV contractor production facilities. This inadequate quality assurance management could increase costs, delay launch schedules, and increase the risk of mission failure.

From ”

Evaluation of the Evolved Expendable Launch Vehicle Program Quality Management System DODIG-2018-045, Department of Defense Office of Inspector General

It is useful to read audit reports and inspection findings from multiple industries. From this we can see trends, make connections and learn.

I see a few things that stand out.

DOD findings

Risk Register

Our evaluation of the RIO database showed that 11 out of 26 risks related to either Atlas V or Delta IV launch vehicle were in “red” status, which indicates that risk mitigation was behind schedule.

It is not enough to identify risks (though that is a critical place to start). You just can’t track them (though again, if you don’t track it you don’t see it). You actually have to have clear plans to mitigate and eliminate the risks. And this is where the program seemed to fall short.

Not a surprise. I think a lot of companies are having these difficulties. In the pharma world the regulatory agencies have been signaling pretty strongly that this is an issue.

Make sure you identify risks, track them, and have plans that are actually carried out to remediate.

Configuration Management

SpaceX failed to comply with AS9100C, section 7.1.3, which requires it to “establish, implement, and maintain a configuration management process.” Configuration management is a controlled process to establish the baseline configuration of a product and any changes to that product. This process should occur during the entire life cycle of a product to provide visibility and control of its physical, functional, and performance attributes.

First rule of reading inspection reports: Things probably went bad if a section starts with standard review 101 material.

That said, hello change management my dear friend.

This was the gist of my ASQ WCQI workshop last May, every industry needs good change management and change control.

Material Management

ULA and AR failed to comply with AS9100C, section 8.3, which requires them to “ensure that product which does not conform to product requirements are identified and controlled to prevent its unintended use or delivery.”

At ULA, we found 18 expired limited-life material items that were between 32 and 992 days past their expiration dates, but available for use on EELV flight hardware. This material should have been impounded and dispositioned. The use of expired limited-life items, such as glues and bonding agents, could result in product that does not meet specifications and may require costly rework.

I find it hard to believe that these companies aren’t tracking inventory. If they are tracking inventory and have any sort of cycle count process then the mechanism exists to ensure expired material is removed from the possibility of use. And yet we still see these observations across the pharma industry as well.

Concluding Thoughts

Quality Management has it its core the same principles, no matter the industry. We use similar tools. Leverage the best practices out there. Read about stresses other companies are having, learn from them and remediate at your own organization.