Designing Level 2 Training Effectiveness Assessments

In the Kilpatrick model, a level 2 assessment measures how much individuals learned. It is asking did the learners actually learn what we wanted them to learn? Did we actually advance knowledge?

For many of us, the old go-to is the multiple-choice quiz.

Measuring Training Effectiveness for Organizational Performance

If we actually want to assess a learner’s ability to do something or think critically about a topic, a multiple-choice quiz isn’t going to work. This isn’t to say that a multiple-choice quiz can’t be challenging, but the focus of a multiple-choice quiz is on the learner’s understanding of the content, not on the learner’s knowledge of how to apply the content to a variety of different contexts.

Say we are designing a root cause analysis course. By the end of the course, your learners should be able to understand some core principles of root cause analysis so that they can perform better investigations, find root causes and determine appropriate CAPAs. While there may be some inherently wrong approaches to root cause analysis that could be assessed in a multiple-choice quiz, a skilled investigator will likely not be dealing with obvious “right” and “wrong” ways to identify causes. Most investigations require complex interactions with people. As such, there may be multiple decisions an investigator needs to make and, within the scope of a course, it could be really hard to identify what skills a budding investigator needs to develop through multiple-choice quizzes alone.

So, what kinds of assessments could you use beyond multiple-choice quizzes and when should you use them? There’s a lot of complexity to these choices which ultimately need to align what
you want people in the course to learn with how you think they can best demonstrate evidence of that learning.

Assessment Instrument	When to use it	Example
Multiple-Choice Quiz or Exam	To assess a learner’s understanding of a concept, definition, or specific process. Could also be used to assess responses or reactions to a scenario-based question if there are clear “right” or “wrong” responses.	Understanding of core concepts of root cause analysis. Simple branching choices, for example what tool to use when.
Open-Ended Questions	To assess a learner’s ability to interpret and apply a new idea. Could also be used to assess a learner’s ability to describe an approach to a process or problem.	Demonstrate knowledge of root cause analysis techniques through various practice exercises.
Long-Form Written Assignment	To assess a learner’s ability to make an argument, analyze a text or current event, or use outside evidence to inform a particular claim. Could also be used to assess a learner’s understanding of how to produce a piece of writing specific to a particular field or discipline (for example, a lab report in a lab sciences context or a policy memo in a public policy context).	Write an analysis and investigation report up from a example.
Project	To assess a learner’s ability to make a new product and apply skills learned to build an independent work. Could also be used to assess a learner’s understanding of how to create a field-specific artifact.	Conduct a root cause analysis from an exercise. On the job training.
Portfolio	To assess a learner’s ability to grow, revise, and create a body of work over a particular period of time.	Review of investigations on periodic basis

Assessment Types

A lot of learning experiences will implement a combination of these types of assessments in a course, and it’s likely that at different phases of your course and for different purposes, you
will need to select more than one assessment or evaluation method.

Remember that an assessment serves two additional purposes: It helps the learners recognize where they are in the course so that they have an understanding of the progress, and it helps you, as the facilitator, see what challenges and triumphs the learners are experiencing all the way throughout the course.

FDA Revised MAPP Impact to Foreign Inspections

The FDA has updated its MAPP for staff, revising the “Understanding CDER’s Risk-Based Site Selection Model” (5014.1 Rev. 1). The update details how the FDA will prioritize inspections under the SSM and adds a risk factor for establishments in countries with a history of violations. This is a new risk factor for a region or entire country, whereas previously, mainly individual criteria such as site quality history, site type, and time since last inspection were used.

This is a fascinating addition, and I’d love to see the actual data on this risk factor. Anyone know if this can be FOIA’d? It would be interesting to see the difference between India and Mexico or even if they’ve subdivided the US.

Good Scientific Practices as Phase Appropriate

There has been increasing evidence in recent years that research in life sciences is lacking in reproducibility and data quality. This raises the need for effective systems to improve data integrity in the evolving non-GxP research environment. Reproducibility is a defining principle of scientific research, and broadly refers to the ability of researchers, other than the original researchers, to achieve the same findings using the same data and analysis data reproducibility is key to the reinforcement and credibility of scientific evidence. All results should be replicable by different investigators in varied geographical settings, using independent data, instruments, and analytical methods.

Some examples:

In 2022 there were 11 Federal Register notices with ORI findings of research misconduct that involved Public Health Service support or funding. These cases included falsified data submitted in National Institutes of Health grant applications and PHS-supported publications. These cases resulted debarment periods of up to four years and supervision periods of up to 12 years.
Novartis “data manipulation” involving its Zolgensma gene therapy
Leen Kawas Resigned as CEO of Athira in 2021 following an investigation into her doctoral work.

Without a doubt it is critical to build a quality culture within our research organizations. Through educating our scientific staff we can continue to innovate and discover new pathways, new drugs and new treatments. Efficient processes enhance research effectiveness and lead to scientific discoveries. Data integrity supports good science, drug safety, products and treatment development for patients and customers. While this looks similar in research as in later phases there are 4 primary pillars:

Train researchers on basic documentation processes and good scientific practices to ensure data integrity and quality. Targeted training should be added on new guidelines, processes and regulations applied to their specific activities.
Empower for change and to speak up
Incentives for Behaviours Which Support Research Quality
Promote a Positive Error Culture

I’m a huge fan of the EQIPD approach:

https://eqipd181379605.wordpress.com/.

Bespalov, A., Bernard, R., Gilis, A., Gerlach, B., Guillén, J., Castagné, V., Lefevre, I. A., Ducrey, F., Monk, L., Bongiovanni, S., Altevogt, B., Arroyo-Araujo, M., Bikovski, L., Bruin, N. de, Castaños-Vélez, E., Dityatev, A., Emmerich, C. H., Fares, R., Ferland-Beckham, C., … Steckler, T. (2021, May 24). Introduction to the EQIPD Quality System. eLife. https://elifesciences.org/articles/63294

Phase Appropriate – An Unpacking

There is no term more misused and misunderstood than “Phase Appropriate.” It is one of those terms that just about everyone involved in FDA-regulated industries has an opinion on and one where we all get tripped up.

What do we mean by phase?

Drug development can be divided into discovery, preclinical studies, clinical development, and market approval.

Each one of these phases is further broken down.

It is also important to remember that certain activities may start in earlier phases. For example, for manufacturing, tech transfer, and commercial manufacturing can start in Phase 3 (and more and more these days even 2!).

A similar approach can apply to medical devices.

The GxPs – a brief definition

Phase Appropriate GMPs

A Review of Regulations

21 CFR 210.2(c)	An investigational drug for use in a phase 1 study, as described in § 312.21(a) of this chapter, is subject to the statutory requirements set forth in 21 U.S.C. 351(a)(2)(B). The production of such drug is exempt from compliance with the regulations in part 211 of this chapter. However, this exemption does not apply to an investigational drug for use in a phase 1 study once the investigational drug has been made available for use by or for the sponsor in a phase 2 or phase 3 study, as described in § 312.21(b) and (c) of this chapter, or the drug has been lawfully marketed. If the investigational drug has been made available in a phase 2 or phase 3 study or the drug has been lawfully marketed, the drug for use in the phase 1 study must comply with part 211.
FDA Guidance	CGMP for Phase 1 Investigational Drugs
EMA/INS/GMP/258937/2022	Guideline on the responsibilities of the sponsor with regard to handling and shipping of investigational medicinal products for human use in accordance with Good Clinical Practice and Good Manufacturing Practice
Eudralex Volume 4 Annex 13	Investigational Medicinal Products

ICH Q10 Diagram of the ICH Q10 Pharmaceutical Quality System Model (Annex 2)

What Activities are Phase-specific for the GMPs

Phase 1:

Critical quality attributes identified with safety Critical Quality Attributes (CQAs) clearly documented
Process changes as information is accumulated
Controls for analytical methods

Phase 2:

Processes characterized and Production and Process Controls (PPC) identified
Analytical methods are qualified
Materials acceptance criteria
Critical vendors qualified

Phase 3:

Processes validated with Production and Process Controls (PPC) identified and controlled
Validation of analytical methods
Materials have been fully qualified and tested upon receipt as appropriate

What About the Quality System?

ICH Q10 clearly spells out the PQS requirements, breaking down into stages of Pharmaceutical Development (usually Phase 1 and earlier), Technology Transfer (usually phase 2), Commercial Manufacturing (which may start before approval) and Product Discontinuation. Q10 then lays out the expectations by these stages for the four key elements of:

Process performance and product quality monitoring system
Corrective action and preventive action (CAPA) system
Change management system
Management review of process performance and product quality.

	Pharmaceutical Development	Technology Transfer	Commercial Manufacturing	Product Discontinuation
Process Performance and Product Quality	Process and product knowledge generated and process and product monitoring conducted throughout development can be used to establish a control strategy for manufacturing.	Monitoring during scale-up activities can provide a preliminary indication of process performance and the successful integration into manufacturing. Knowledge obtained during transfer and scale up activities can be useful in further developing the control strategy.	A well-defined system for process performance and product quality monitoring should be applied to assure performance within a state of control and to identify improvement areas.	Once manufacturing ceases, monitoring such as stability testing should continue to completion of the studies. Appropriate action on marketed product should continue to be executed according to regional regulations.
Corrective Action and Preventive Action	Product or process variability is explored. CAPA methodology is useful where corrective actions and preventive actions are incorporated into the iterative design and development process.	CAPA can be used as an effective system for feedback, feedforward and continual improvement.	CAPA should be used and the effectiveness of the actions should be evaluated.	CAPA should continue after the product is discontinued. The impact on product remaining on the market should be considered as well as other products which might be impacted.
Change Management	Change is an inherent part of the development process and should be documented; the formality of the change management process should be consistent with the stage of pharmaceutical development.	The change management system should provide management and documentation of adjustments made to the process during technology transfer activities.	A formal change management system should be in place for commercial manufacturing. Oversight by the quality unit should provide assurance of appropriate science and risk based assessments.	Any changes after product discontinuation should go through an appropriate change management system.
Management Review of Process Performance and Product Quality	Aspects of management review can be performed to ensure adequacy of the product and process design.	Aspects of management review should be performed to ensure the developed product and process can be manufactured at commercial scale.	Management review should be a structured system, as described above, and should support continual improvement.	Management review should include such items as product stability and product quality complaints.

ICH Stage appropriate quality system elements

Together with ICH Q9, this sets forth a framework of building knowledge and risk management into all aspects of the system together with a robust issue management mindset. There are really three things driving this.

Consistency in execution
Document decision making
Follow through

Some aspects remain pretty steady in all phases/stages, while others will grow as the organization develops.

The Difference Between Maturity and Phase Appropriate

People confuse phase appropriate with maturity all the time. Phase appropriate means doing the right activities in the right order. Maturity means the how is the most effective possible.

Quality Management Maturity (QMM) is the state attained when drug manufacturers have consistent, reliable, and robust business processes to achieve quality objectives and promote continual improvement. This is both composed of phase independent and phase dependent aspects.

Remember, a Quality Culture is the foundation that makes the rest of this happen.