Designing Level 2 Training Effectiveness Assessments

In the Kilpatrick model, a level 2 assessment measures how much individuals learned. It is asking did the learners actually learn what we wanted them to learn? Did we actually advance knowledge?

For many of us, the old go-to is the multiple-choice quiz.

Measuring Training Effectiveness for Organizational Performance

If we actually want to assess a learner’s ability to do something or think critically about a topic, a multiple-choice quiz isn’t going to work. This isn’t to say that a multiple-choice quiz can’t be challenging, but the focus of a multiple-choice quiz is on the learner’s understanding of the content, not on the learner’s knowledge of how to apply the content to a variety of different contexts.

Say we are designing a root cause analysis course. By the end of the course, your learners should be able to understand some core principles of root cause analysis so that they can perform better investigations, find root causes and determine appropriate CAPAs. While there may be some inherently wrong approaches to root cause analysis that could be assessed in a multiple-choice quiz, a skilled investigator will likely not be dealing with obvious “right” and “wrong” ways to identify causes. Most investigations require complex interactions with people. As such, there may be multiple decisions an investigator needs to make and, within the scope of a course, it could be really hard to identify what skills a budding investigator needs to develop through multiple-choice quizzes alone.

So, what kinds of assessments could you use beyond multiple-choice quizzes and when should you use them? There’s a lot of complexity to these choices which ultimately need to align what
you want people in the course to learn with how you think they can best demonstrate evidence of that learning.

Assessment Instrument	When to use it	Example
Multiple-Choice Quiz or Exam	To assess a learner’s understanding of a concept, definition, or specific process. Could also be used to assess responses or reactions to a scenario-based question if there are clear “right” or “wrong” responses.	Understanding of core concepts of root cause analysis. Simple branching choices, for example what tool to use when.
Open-Ended Questions	To assess a learner’s ability to interpret and apply a new idea. Could also be used to assess a learner’s ability to describe an approach to a process or problem.	Demonstrate knowledge of root cause analysis techniques through various practice exercises.
Long-Form Written Assignment	To assess a learner’s ability to make an argument, analyze a text or current event, or use outside evidence to inform a particular claim. Could also be used to assess a learner’s understanding of how to produce a piece of writing specific to a particular field or discipline (for example, a lab report in a lab sciences context or a policy memo in a public policy context).	Write an analysis and investigation report up from a example.
Project	To assess a learner’s ability to make a new product and apply skills learned to build an independent work. Could also be used to assess a learner’s understanding of how to create a field-specific artifact.	Conduct a root cause analysis from an exercise. On the job training.
Portfolio	To assess a learner’s ability to grow, revise, and create a body of work over a particular period of time.	Review of investigations on periodic basis

Assessment Types

A lot of learning experiences will implement a combination of these types of assessments in a course, and it’s likely that at different phases of your course and for different purposes, you
will need to select more than one assessment or evaluation method.

Remember that an assessment serves two additional purposes: It helps the learners recognize where they are in the course so that they have an understanding of the progress, and it helps you, as the facilitator, see what challenges and triumphs the learners are experiencing all the way throughout the course.

Three things that get in the way of good training

Culture of Delivery

Does training in your organization seem like death by PowerPoint? Is learning viewed as something an expert dumps in the lap of the learner.? However, that’s not what learning is – lectures and one-way delivery end up resulting in very little learning.

For deeper meaning to occur, invest in professionally facilitated experiences that enable staff to form mental models they remember. Get people thinking before and after the training to ensure that the mental model stays fresh in the mind.

Culture of Cutting Time

Avoid the desire for training in shorter and shorter chunks. The demands of the workplace are increasingly complex and stressful, so any time out of the office is a serious cost. The paradox is that by shortening the training, we don’t give the time for structured learning, which sabotages the investment when the training program could be substantially improved by adding the time to allow the learning to be consolidated.

Culture of Un-Fun

We know that learning takes place when people have fun, stress is low, and the environment encourages discovery. Make training cheerful and open rather than dull and quiet. Encourage lots of informal learning opportunities. Give more control to the learner to shape their experience. Have fun!

Building a learning culture

Our organizations are either growing or they’re dying. The key thing that drives growth in organizations is when their employees are learning. To strengthen our organizations, our teams, ourselves we need to ensure our culture allows people to be exposed to new and challenging opportunities to learn.

We learn constantly. Most of that learning, however, is incremental, improvements that build on what we already know and do. We expand our knowledge and refine our skills in ways that strengthen our identities and commitments. This process sharpens competence and broadens expertise, and is key in building subject matter experts.

Incremental learning can allow people to grow in a workplace until they reach the limit on their resources for new learning – think of it as an S-curve. Eventually, there isn’t enough opportunities to learn. Furthermore, learning that broadens our expertise is valuable, but it is not enough. Incremental learning does not alter the way we see others, the world, and ourselves.

The second type of learning is called transformative, it changes our perspectives laying the foundations for growth and innovative leaps.

Both kinds of learning are necessary. Incremental learning helps us deliver, while transformative learning helps us develop. Both are necessary, but too often we allow incremental learning to be haphazard and make no space for transformative learning.

In both cases we need to build spaces to drive learning.

We often see incremental in our training programs, while transformative is critical for culture building.

	Incremental Learning	Transformative Learning
Good for	Knowledge and Skills	Purpose and Presence
Source of Learning	Experts (models)	Experience (moments)
Work required	Deliberate Practice	Reflective engagement
Aim of process	New action (a better way)	New meaning (a better why)
Role of others	Focusing practice	Inviting Interpretation

Key aspects of the two styles

Bibliography

References

Bersin. (2018, July 08). A new paradigm for corporate training: Learning in the flow of work. Retrieved December 31, 2019, from https://joshbersin.com/2018/06/a-new-paradigm-for-corporate-training-learning-in-the-flow-of-work/
Boyatzis E., & Akrivou, K. (2006). The ideal self as the driver of intentional change. Journal of Management Development, 25(7), 624-642. doi:10.1108/02621710610678454
Brown D., & Starkey, K. (2000). Organizational identity and learning: A psychodynamic perspective. The Academy of Management Review, 25(1), 102. doi:10.2307/259265
Hoffman, R., Yeh, C., & Casnocha, B. (2019). Learn from people, not classes. Harvard Business Review, 97(3). Retrieved December 31, 2019, from https://hbr.org/2019/03/educating-the-next-generation-of-leaders
Mezirow, J. (1991). Transformative dimensions of adult learning. San Francisco, CA: Jossey-Bass.
Petriglieri, G., Petriglieri, J. L., & Wood, J. D. (2017). Fast tracks and Inner Journeys: Crafting Portable selves for contemporary careers. Administrative Science Quarterly, 63(3), 479-525. doi:10.1177/0001839217720930