In the spirit of disclosure, I may have referenced this one once or twice, but I think it is essential to know exactly what the research says when we talk about deliberate practice, a topic I end to be passionate about.
Scott Young “The 10,000-Hour Rule is a Myth.”
ISO 31000-2018 “Risk Management Guidelines” discusses on-going monitoring and review of risk management activities. We see a similar requirement in ICH Q9(r1) for the pharmaceutical industry. In many organizations we can take a lot of time on the performance of risk assessments (hopefully effectively) and a lot of time mitigating risks (again, hopefully effectively) but many organizations struggle in maintaining a lifecycle approach.
To do appropriate lifecycle management we should ensure three things:
Reviews are a critical part of our risk management process framework.
This living risk management approach effectively drives work in Control Environment, Response and Stress Testing.
At heart lies the ongoing connection between risk management and knowledge management.
A firm requirement throughout the GxP regulations and the various ISO standards is that individuals are appropriately trained and qualified to do their work.
Inevitably, that appropriately trained and qualified comes down to the trainer who is conducting the training. How do we train our trainers to ensure that individuals are learning and acquiring all of the skills, knowledge, and insight they need to perform their roles well.
Inadequate training is a consistent finding across the GxPs, so you have to ask are we training our trainers to an appropriate level to make the training effective? Equally important you are spending a lot of time and money training people so you want it to be effective and worth the resources spent.
There are really two options for trainers: 1. Trainers who become qualified to teach a course and 2. SMEs who are qualified to be trainers. In either case, you need that qualification mechanism to ensure your trainers can train. I’ll be honest for technical material I prefer SMEs being trained to be trainers as the experience is usually a whole lot better for the trainee.
This training focuses on being able to deliver informal and formal learning solutions in a manner that is both engaging and effective. Be able to:
This qualification path will prove itself valuable.Through this we can ensure that our trainings meet their four objectives and that participants can demonstrate:
Being a trainer is critical for true subject matter expertise and process ownership.
And as an aside, notice I didn’t include instructional design, this is where your training team can really add value!
In the Kilpatrick model, a level 2 assessment measures how much individuals learned. It is asking did the learners actually learn what we wanted them to learn? Did we actually advance knowledge?
For many of us, the old go-to is the multiple-choice quiz.
If we actually want to assess a learner’s ability to do something or think critically about a topic, a multiple-choice quiz isn’t going to work. This isn’t to say that a multiple-choice quiz can’t be challenging, but the focus of a multiple-choice quiz is on the learner’s understanding of the content, not on the learner’s knowledge of how to apply the content to a variety of different contexts.
Say we are designing a root cause analysis course. By the end of the course, your learners should be able to understand some core principles of root cause analysis so that they can perform better investigations, find root causes and determine appropriate CAPAs. While there may be some inherently wrong approaches to root cause analysis that could be assessed in a multiple-choice quiz, a skilled investigator will likely not be dealing with obvious “right” and “wrong” ways to identify causes. Most investigations require complex interactions with people. As such, there may be multiple decisions an investigator needs to make and, within the scope of a course, it could be really hard to identify what skills a budding investigator needs to develop through multiple-choice quizzes alone.
So, what kinds of assessments could you use beyond multiple-choice quizzes and when should you use them? There’s a lot of complexity to these choices which ultimately need to align what
you want people in the course to learn with how you think they can best demonstrate evidence of that learning.
| Assessment Instrument | When to use it | Example |
| Multiple-Choice Quiz or Exam | To assess a learner’s understanding of a concept, definition, or specific process. Could also be used to assess responses or reactions to a scenario-based question if there are clear “right” or “wrong” responses. | Understanding of core concepts of root cause analysis. Simple branching choices, for example what tool to use when. |
| Open-Ended Questions | To assess a learner’s ability to interpret and apply a new idea. Could also be used to assess a learner’s ability to describe an approach to a process or problem. | Demonstrate knowledge of root cause analysis techniques through various practice exercises. |
| Long-Form Written Assignment | To assess a learner’s ability to make an argument, analyze a text or current event, or use outside evidence to inform a particular claim. Could also be used to assess a learner’s understanding of how to produce a piece of writing specific to a particular field or discipline (for example, a lab report in a lab sciences context or a policy memo in a public policy context). | Write an analysis and investigation report up from a example. |
| Project | To assess a learner’s ability to make a new product and apply skills learned to build an independent work. Could also be used to assess a learner’s understanding of how to create a field-specific artifact. | Conduct a root cause analysis from an exercise. On the job training. |
| Portfolio | To assess a learner’s ability to grow, revise, and create a body of work over a particular period of time. | Review of investigations on periodic basis |
A lot of learning experiences will implement a combination of these types of assessments in a course, and it’s likely that at different phases of your course and for different purposes, you
will need to select more than one assessment or evaluation method.
Remember that an assessment serves two additional purposes: It helps the learners recognize where they are in the course so that they have an understanding of the progress, and it helps you, as the facilitator, see what challenges and triumphs the learners are experiencing all the way throughout the course.
There is no term more misused and misunderstood than “Phase Appropriate.” It is one of those terms that just about everyone involved in FDA-regulated industries has an opinion on and one where we all get tripped up.
Drug development can be divided into discovery, preclinical studies, clinical development, and market approval.
Each one of these phases is further broken down.
It is also important to remember that certain activities may start in earlier phases. For example, for manufacturing, tech transfer, and commercial manufacturing can start in Phase 3 (and more and more these days even 2!).
A similar approach can apply to medical devices.
| 21 CFR 210.2(c) | An investigational drug for use in a phase 1 study, as described in § 312.21(a) of this chapter, is subject to the statutory requirements set forth in 21 U.S.C. 351(a)(2)(B). The production of such drug is exempt from compliance with the regulations in part 211 of this chapter. However, this exemption does not apply to an investigational drug for use in a phase 1 study once the investigational drug has been made available for use by or for the sponsor in a phase 2 or phase 3 study, as described in § 312.21(b) and (c) of this chapter, or the drug has been lawfully marketed. If the investigational drug has been made available in a phase 2 or phase 3 study or the drug has been lawfully marketed, the drug for use in the phase 1 study must comply with part 211. |
| FDA Guidance | CGMP for Phase 1 Investigational Drugs |
| EMA/INS/GMP/258937/2022 | Guideline on the responsibilities of the sponsor with regard to handling and shipping of investigational medicinal products for human use in accordance with Good Clinical Practice and Good Manufacturing Practice |
| Eudralex Volume 4 Annex 13 | Investigational Medicinal Products |
ICH Q10 clearly spells out the PQS requirements, breaking down into stages of Pharmaceutical Development (usually Phase 1 and earlier), Technology Transfer (usually phase 2), Commercial Manufacturing (which may start before approval) and Product Discontinuation. Q10 then lays out the expectations by these stages for the four key elements of:
| Pharmaceutical Development | Technology Transfer | Commercial Manufacturing | Product Discontinuation | |
| Process Performance and Product Quality | Process and product knowledge generated and process and product monitoring conducted throughout development can be used to establish a control strategy for manufacturing. | Monitoring during scale-up activities can provide a preliminary indication of process performance and the successful integration into manufacturing. Knowledge obtained during transfer and scale up activities can be useful in further developing the control strategy. | A well-defined system for process performance and product quality monitoring should be applied to assure performance within a state of control and to identify improvement areas. | Once manufacturing ceases, monitoring such as stability testing should continue to completion of the studies. Appropriate action on marketed product should continue to be executed according to regional regulations. |
| Corrective Action and Preventive Action | Product or process variability is explored. CAPA methodology is useful where corrective actions and preventive actions are incorporated into the iterative design and development process. | CAPA can be used as an effective system for feedback, feedforward and continual improvement. | CAPA should be used and the effectiveness of the actions should be evaluated. | CAPA should continue after the product is discontinued. The impact on product remaining on the market should be considered as well as other products which might be impacted. |
| Change Management | Change is an inherent part of the development process and should be documented; the formality of the change management process should be consistent with the stage of pharmaceutical development. | The change management system should provide management and documentation of adjustments made to the process during technology transfer activities. | A formal change management system should be in place for commercial manufacturing. Oversight by the quality unit should provide assurance of appropriate science and risk based assessments. | Any changes after product discontinuation should go through an appropriate change management system. |
| Management Review of Process Performance and Product Quality | Aspects of management review can be performed to ensure adequacy of the product and process design. | Aspects of management review should be performed to ensure the developed product and process can be manufactured at commercial scale. | Management review should be a structured system, as described above, and should support continual improvement. | Management review should include such items as product stability and product quality complaints. |
Together with ICH Q9, this sets forth a framework of building knowledge and risk management into all aspects of the system together with a robust issue management mindset. There are really three things driving this.
Some aspects remain pretty steady in all phases/stages, while others will grow as the organization develops.
People confuse phase appropriate with maturity all the time. Phase appropriate means doing the right activities in the right order. Maturity means the how is the most effective possible.
Quality Management Maturity (QMM) is the state attained when drug manufacturers have consistent, reliable, and robust business processes to achieve quality objectives and promote continual improvement. This is both composed of phase independent and phase dependent aspects.
Remember, a Quality Culture is the foundation that makes the rest of this happen.
