ISO 31000-2018 “Risk Management Guidelines” discusses on-going monitoring and review of risk management activities. We see a similar requirement in ICH Q9(r1) for the pharmaceutical industry. In many organizations we can take a lot of time on the performance of risk assessments (hopefully effectively) and a lot of time mitigating risks (again, hopefully effectively) but many organizations struggle in maintaining a lifecycle approach.
To do appropriate lifecycle management we should ensure three things:
A firm requirement throughout the GxP regulations and the various ISO standards is that individuals are appropriately trained and qualified to do their work.
Inevitably, that appropriately trained and qualified comes down to the trainer who is conducting the training. How do we train our trainers to ensure that individualsare learning and acquiring all of the skills, knowledge, and insight they need to perform their roles well.
Inadequate training is a consistent finding across the GxPs, so you have to ask are we training our trainers to an appropriate level to make the training effective? Equally important you are spending a lot of time and money training people so you want it to be effective and worth the resources spent.
There are really two options for trainers: 1. Trainers who become qualified to teach a course and 2. SMEs who are qualified to be trainers. In either case, you need that qualification mechanism to ensure your trainers can train. I’ll be honest for technical material I prefer SMEs being trained to be trainers as the experience is usually a whole lot better for the trainee.
This training focuses on being able to deliver informal and formal learning solutions in a manner that is both engaging and effective. Be able to:
Manage the learning environment.
Prepare for training delivery.
Convey objectives.
Align learning solutions with course objectives and learner needs.
Establish credibility as an instructor.
Create a positive learning climate.
Deliver various learning methodologies.
Facilitate learning.
Encourage participation and build learner motivation.
Deliver constructive feedback.
Ensure learning outcomes.
Evaluate solutions.
This qualification path will prove itself valuable.Through this we can ensure that our trainings meet their four objectives and that participants can demonstrate:
Awareness: Participant says, “I’ve heard that!”
Understanding: Participant recognizes the subject matter and then explains it.
Practice: Participant actually uses the learning on the job.
Mastery: Participant can use the acquired knowledge to teach others.
In the Kilpatrick model, a level 2 assessment measures how much individuals learned. It is asking did the learners actually learn what we wanted them to learn? Did we actually advance knowledge?
For many of us, the old go-to is the multiple-choice quiz.
If we actually want to assess a learner’s ability to do something or think critically about a topic, a multiple-choice quiz isn’t going to work. This isn’t to say that a multiple-choice quiz can’t be challenging, but the focus of a multiple-choice quiz is on the learner’s understanding of the content, not on the learner’s knowledge of how to apply the content to a variety of different contexts.
Say we are designing a root cause analysis course. By the end of the course, your learners should be able to understand some core principles of root cause analysis so that they can perform better investigations, find root causes and determine appropriate CAPAs. While there may be some inherently wrong approaches to root cause analysis that could be assessed in a multiple-choice quiz, a skilled investigator will likely not be dealing with obvious “right” and “wrong” ways to identify causes. Most investigations require complex interactions with people. As such, there may be multiple decisions an investigator needs to make and, within the scope of a course, it could be really hard to identify what skills a budding investigator needs to develop through multiple-choice quizzes alone.
So, what kinds of assessments could you use beyond multiple-choice quizzes and when should you use them? There’s a lot of complexity to these choices which ultimately need to align what you want people in the course to learn with how you think they can best demonstrate evidence of that learning.
Assessment Instrument
When to use it
Example
Multiple-Choice Quiz or Exam
To assess a learner’s understanding of a concept, definition, or specific process. Could also be used to assess responses or reactions to a scenario-based question if there are clear “right” or “wrong” responses.
Understanding of core concepts of root cause analysis. Simple branching choices, for example what tool to use when.
Open-Ended Questions
To assess a learner’s ability to interpret and apply a new idea. Could also be used to assess a learner’s ability to describe an approach to a process or problem.
Demonstrate knowledge of root cause analysis techniques through various practice exercises.
Long-Form Written Assignment
To assess a learner’s ability to make an argument, analyze a text or current event, or use outside evidence to inform a particular claim. Could also be used to assess a learner’s understanding of how to produce a piece of writing specific to a particular field or discipline (for example, a lab report in a lab sciences context or a policy memo in a public policy context).
Write an analysis and investigation report up from a example.
Project
To assess a learner’s ability to make a new product and apply skills learned to build an independent work. Could also be used to assess a learner’s understanding of how to create a field-specific artifact.
Conduct a root cause analysis from an exercise.
On the job training.
Portfolio
To assess a learner’s ability to grow, revise, and create a body of work over a particular period of time.
Review of investigations on periodic basis
Assessment Types
A lot of learning experiences will implement a combination of these types of assessments in a course, and it’s likely that at different phases of your course and for different purposes, you will need to select more than one assessment or evaluation method.
Remember that an assessment serves two additional purposes: It helps the learners recognize where they are in the course so that they have an understanding of the progress, and it helps you, as the facilitator, see what challenges and triumphs the learners are experiencing all the way throughout the course.
There is no term more misused and misunderstood than “Phase Appropriate.” It is one of those terms that just about everyone involved in FDA-regulated industries has an opinion on and one where we all get tripped up.
What do we mean by phase?
Drug development can be divided into discovery, preclinical studies, clinical development, and market approval.
Each one of these phases is further broken down.
It is also important to remember that certain activities may start in earlier phases. For example, for manufacturing, tech transfer, and commercial manufacturing can start in Phase 3 (and more and more these days even 2!).
An investigational drug for use in a phase 1 study, as described in § 312.21(a) of this chapter, is subject to the statutory requirements set forth in 21 U.S.C. 351(a)(2)(B). The production of such drug is exempt from compliance with the regulations in part 211 of this chapter. However, this exemption does not apply to an investigational drug for use in a phase 1 study once the investigational drug has been made available for use by or for the sponsor in a phase 2 or phase 3 study, as described in § 312.21(b) and (c) of this chapter, or the drug has been lawfully marketed. If the investigational drug has been made available in a phase 2 or phase 3 study or the drug has been lawfully marketed, the drug for use in the phase 1 study must comply with part 211.
Guideline on the responsibilities of the sponsor with regard to handling and shipping of investigational medicinal products for human use in accordance with Good Clinical Practice and Good Manufacturing Practice
Processes characterized and Production and Process Controls (PPC) identified
Analytical methods are qualified
Materials acceptance criteria
Critical vendors qualified
Phase 3:
Processes validated with Production and Process Controls (PPC) identified and controlled
Validation of analytical methods
Materials have been fully qualified and tested upon receipt as appropriate
What About the Quality System?
ICH Q10 clearly spells out the PQS requirements, breaking down into stages of Pharmaceutical Development (usually Phase 1 and earlier), Technology Transfer (usually phase 2), Commercial Manufacturing (which may start before approval) and Product Discontinuation. Q10 then lays out the expectations by these stages for the four key elements of:
Process performance and product quality monitoring system
Corrective action and preventive action (CAPA) system
Change management system
Management review of process performance and product quality.
Pharmaceutical Development
Technology Transfer
Commercial Manufacturing
Product Discontinuation
Process Performance and Product Quality
Process and product knowledge generated and process and product monitoring conducted throughout development can be used to establish a control strategy for manufacturing.
Monitoring during scale-up activities can provide a preliminary indication of process performance and the successful integration into manufacturing. Knowledge obtained during transfer and scale up activities can be useful in further developing the control strategy.
A well-defined system for process performance and product quality monitoring should be applied to assure performance within a state of control and to identify improvement areas.
Once manufacturing ceases, monitoring such as stability testing should continue to completion of the studies. Appropriate action on marketed product should continue to be executed according to regional regulations.
Corrective Action and Preventive Action
Product or process variability is explored. CAPA methodology is useful where corrective actions and preventive actions are incorporated into the iterative design and development process.
CAPA can be used as an effective system for feedback, feedforward and continual improvement.
CAPA should be used and the effectiveness of the actions should be evaluated.
CAPA should continue after the product is discontinued. The impact on product remaining on the market should be considered as well as other products which might be impacted.
Change Management
Change is an inherent part of the development process and should be documented; the formality of the change management process should be consistent with the stage of pharmaceutical development.
The change management system should provide management and documentation of adjustments made to the process during technology transfer activities.
A formal change management system should be in place for commercial manufacturing. Oversight by the quality unit should provide assurance of appropriate science and risk based assessments.
Any changes after product discontinuation should go through an appropriate change management system.
Management Review of Process Performance and Product Quality
Aspects of management review can be performed to ensure adequacy of the product and process design.
Aspects of management review should be performed to ensure the developed product and process can be manufactured at commercial scale.
Management review should be a structured system, as described above, and should support continual improvement.
Management review should include such items as product stability and product quality complaints.
ICH Stage appropriate quality system elements
Together with ICH Q9, this sets forth a framework of building knowledge and risk management into all aspects of the system together with a robust issue management mindset. There are really three things driving this.
Consistency in execution
Document decision making
Follow through
Some aspects remain pretty steady in all phases/stages, while others will grow as the organization develops.
The Difference Between Maturity and Phase Appropriate
People confuse phase appropriate with maturity all the time. Phase appropriate means doing the right activities in the right order. Maturity means the how is the most effective possible.
Quality Management Maturity (QMM) is the state attained when drug manufacturers have consistent, reliable, and robust business processes to achieve quality objectives and promote continual improvement. This is both composed of phase independent and phase dependent aspects.
Remember, a Quality Culture is the foundation that makes the rest of this happen.
With ICH E6(r3) in draft, I think it is important to look at the current state of risk management expectations in a clinical study.
Risk management is an essential part of any clinical study, and is a critical component of the ICH E6 and E8 guidelines for Good Clinical Practice (GCP). These guidelines provide a framework for ensuring the safety and well-being of study participants, as well as the integrity and reliability of the study data. By following the principles outlined in these guidelines, researchers can help to ensure that their study results are reliable and can be used to inform clinical practice.
Through risk management we ensure the four main goals of the GCPs are obtained.
The ICH E6 guideline provides recommendations for the conduct of clinical trials, emphasizing the importance of risk management, specifying that a risk management plan should be developed and implemented for each study. The guideline also provides recommendations for the content of the risk management plan, including the identification of potential risks, the assessment of their likelihood and potential impact, and the development of strategies for managing or mitigating those risks.
Risk management is a key enabler and result of the quality management system.
The ICH E8 guideline, which focuses on the conduct of clinical trials also emphasizes the importance of risk management. It specifies that the risk management plan should include a comprehensive evaluation of the risks associated with the study interventions, as well as a plan for managing or mitigating those risks. The guideline also recommends that the risk management plan be regularly reviewed and updated as needed, to ensure that it continues to effectively address the risks facing the study.
When planning a clinical study, sponsors must carefully consider the potential risks involved and take steps to minimize them. Sources of the risk assessment include performing a thorough literature review to identify any known risks associated with the study interventions, as well as conducting pre-study assessments to identify potential risks specific to the study population. E8 also state sthe importance of a wide variety of stakeholders, including the patient population.
Once the study is underway, it’s important to closely monitor for potential risks and have a plan in place for managing them.
In addition to protecting the safety of study participants, effective risk management is also essential for maintaining the integrity of the data being collected. Risks to the study data might include things like errors in data entry or missing data, which can compromise the validity of the study results. To address these risks, sponsors must have robust quality control measures in place, such as regular data audits and checks for missing or inconsistent data.
Overall, the role of risk management in a clinical study is to ensure the safety and well-being of study participants, while also protecting the integrity of the data being collected. By carefully considering and managing potential risks, researchers can help to ensure that their study results are reliable and can be used to inform clinical practice.
Risk Based Monitoring
Risk-based monitoring is a approach to monitoring the quality of a clinical study that focuses on identifying and addressing potential risks to the study. This approach involves regularly assessing the risks associated with a study and implementing strategies to manage or mitigate those risks.
In a risk-based monitoring approach, the study team typically uses a risk register to identify and assess potential risks to the study, such as the potential for errors in data collection or analysis, or the potential for adverse events in study participants. The team then develops a plan for addressing these risks, which might involve implementing additional quality control measures or training for study staff.
During the study, the team regularly monitors for potential risks and takes action to address them as needed. This might involve conducting regular audits or reviews of the study data to identify potential errors, or monitoring the health and well-being of study participants to identify and address any adverse events.
Overall, the goal of risk-based monitoring is to ensure the quality and integrity of a clinical study by proactively identifying and addressing potential risks. By using a risk-based approach, the study team can help to ensure that the study results are reliable and can be used to inform clinical practice.
Risk Register
A risk register is a document that is used to identify, assess, and track potential risks in a clinical study. It typically includes a list of identified risks, along with information about their likelihood and potential impact, as well as the actions that are being taken to manage or mitigate the risks.
In a clinical study, a risk register might include risks such as the potential for errors in data collection or analysis, the potential for adverse events in study participants, or the potential for the study to be impacted by external factors, such as changes in regulatory requirements.
The purpose of a risk register in a clinical study is to help the study team identify and prioritize potential risks, and to develop strategies for addressing them. By having a clear and comprehensive overview of the risks that a study is facing, the team can take proactive steps to manage or mitigate those risks, and can monitor their progress over time.
Overall, a risk register is an essential tool for managing risks in a clinical study. By providing a clear and comprehensive overview of potential risks, it helps the study team identify and address risks in a proactive and effective way.
Identifying potential risks: The first step in implementing a clinical risk management program is to identify potential risks to the study, such as the potential for errors in data collection or analysis, or the potential for adverse events in study participants. This might involve reviewing the study protocol and data collection tools, consulting with the study team and other stakeholders, and conducting a thorough assessment of the study environment.
Assessing risks: Once potential risks have been identified, the next step is to assess their likelihood and potential impact. This will help to prioritize the risks and determine the appropriate level of response. For example, a risk with a high likelihood and a high potential impact might require more immediate action, while a risk with a low likelihood and a low potential impact might not require as much attention.
Developing strategies for managing risks: Based on the assessment of risks, the next step is to develop strategies for managing or mitigating those risks. This might involve implementing additional quality control measures, providing training to study staff, or conducting regular audits or reviews of the study data. The goal is to develop a comprehensive and effective plan for addressing the identified risks.
Monitoring for potential risks: Once the risk management plan is in place, it’s important to regularly monitor for potential risks and take action to address them as needed. This might involve conducting regular audits or reviews of the study data, or monitoring the health and well-being of study participants. By proactively monitoring for potential risks, the study team can help to ensure the safety and well-being of study participants, as well as the integrity and reliability of the study data.
Follow-up and corrective action: If potential risks are identified during the study, it’s important to take prompt action to address them. This might involve implementing corrective action plans, such as retraining study staff or revising the study protocol. It’s also important to track the progress of these plans and ensure that they are effective in addressing the identified risks. By taking timely and effective action to address potential risks, the study team can help to ensure the safety and well-being of study participants, as well as the integrity and reliability of the study data.
Risk Management in the Clinical Study Process
To summarize, each clinical study should:
Identify Risks
Before the study begins, the sponsor should perform a thorough review of the study protocol, data collection tools, and other study-related documents to identify potential risks to the study.
The cross-functional study team, CROs and other relevant stakeholders, such as the sponsor and regulatory authorities, to identify additional potential risks.
All identified risks should be documented in the study’s risk register.
2. Assess Risks
For each identified risk, assess its likelihood and potential impact on the study.
The risks should be prioritized based on their likelihood and potential impact, with a focus on the highest-priority risks.
3. Manage Risks
For each identified risk, the sponsor should develop a plan for managing or mitigating the risk. This plan should be documented in the study’s risk register.
The plan for managing or mitigating each risk should include specific actions to be taken, as well as the individuals or groups responsible for implementing those actions.
4. Monitor Risks
Regularly monitor key risk indicators and the study for success of the study risk plan and to identify new potential risks and take action to address them as needed. This might involve conducting regular audits or reviews of the study data, or monitoring the health and well-being of study participants.
Any significant risks that arise during the study should be reported to the sponsor and relevant regulatory authorities.
Investigations of a Dog 