Measuring Training Effectiveness for Organizational Performance

When designing training we want to make sure four things happen:

  • Training is used correctly as a solution to a performance problem
  • Training has the the right content, objectives or methods
  • Trainees are sent to training for which they do have the basic skills, prerequisite skills, or confidence needed to learn
  • Training delivers the expected learning

Training is a useful lever in organization change and improvement. We want to make sure the training drives organization metrics. And like everything, you need to be able to measure it to improve.

The Kirkpatrick model is a simple and fairly accurate way to measure the effectiveness of adult learning events (i.e., training), and while other methods are introduced periodically, the Kirkpatrick model endures because of its simplicity. The model consists of four levels, each designed to measure a specific element of the training. Created by Donald Kirkpatrick, this model has been in use for over 50 years, evolving over multiple decades through application by learning and development professionals around the world. It is the most recognized method of evaluating the effectiveness of training programs. The model has stood the test of time and became popular due to its ability to break down complex subject into manageable levels. It takes into account any style of training, both informal and formal.

Level 1: Reaction

Kirkpatrick’s first level measures the learners’ reaction to the training. A level 1 evaluation is leveraging the strong correlation between learning retention and how much the learners enjoyed the time spent and found it valuable. Level 1 evaluations, euphemistically called a “smile sheet” should delve deeper than merely whether people liked the course. A good course evaluation will concentrate on three elements: course content, the physical environment and the instructor’s presentation/skills.

Level 2: Learning

Level 2 of Kirkpatrick’s model, learning, measures how much of the content attendees learned as a result of the training session. The best way to make this evaluation is through the use of a pre- and posttest. Pre- and posttests are key to ascertaining whether the participants learned anything in the learning event. Identical pre- and posttests are essential because the difference between the pre- and posttest scores indicates the amount of learning that took place. Without a pretest, one does not know if the trainees knew the material before the session, and unless the questions are the same, one cannot be certain that trainees learned the material in the session.

Level 3: Behavior

Level 3 measures whether the learning is transferred into practice in the workplace.

Level 4: Results

Measures the effect on the business environment. Do we meet objectives?

Evaluation LevelCharacteristicsExamples
Level 1: ReactionReaction evaluation is how the delegates felt, and their personal reactions to the training or learning experience, for example: ▪ Did trainee consider the training relevant?
▪ Did they like the venue, equipment, timing, domestics, etc?
▪ Did the trainees like and enjoy the training?
▪ Was it a good use of their time?
▪ Level of participation
▪ Ease and comfort of experience
▪ feedback forms based on subjective personal reaction to the training experience
▪ Verbal reaction which can be analyzed
▪ Post-training surveys or questionnaires
▪ Online evaluation or grading by delegates
▪ Subsequent verbal or written reports given by delegates to managers back at their jobs
▪ typically ‘happy sheets’
Level 2: LearningLearning evaluation is the measurement of the increase in knowledge or intellectual capability from before to after the learning experience:
▪ Did the trainees learn what intended to be taught?
▪ Did the trainee experience what was intended for them to experience?
▪ What is the extent of advancement or change in the trainees after the training, in the direction or area that was intended?
▪ Interview or observation can be used before and after although it is time-consuming and can be inconsistent
▪ Typically assessments or tests before and after the training
▪ Methods of assessment need to be closely related to the aims of the learning
▪ Reliable, clear scoring and measurements need to be established
▪ hard-copy, electronic, online or interview style assessments are all possible
Level 3: BehaviorBehavior evaluation is the extent to which the trainees applied the learning and changed their behavior, and this can be immediately and several months after the training, depending on the situation:
▪ Did the trainees put their learning into effect when back on the job?
▪ Were the relevant skills and knowledge used?
▪ Was there noticeable and measurable change in the activity and performance of the trainees when back in their roles?
▪ Would the trainee be able to transfer their learning to another person? is the trainee aware of their change in behavior, knowledge, skill level?
▪ Was the change in behavior and new level of knowledge sustained?
▪ Observation and interview over time are required to assess change, relevance of change, and sustainability of change
▪ Assessments need to be designed to reduce subjective judgment of the observer
▪ 360-degree feedback is useful method and need not be used before training, because respondents can make a judgment as to change after training, and this can be analyzed for groups of respondents and trainees
▪ Online and electronic assessments are more difficult to incorporate – assessments tend to be more successful when integrated within existing management and coaching protocols
Level 4: ResultsResults evaluation is the effect on the business or environment resulting from the improved performance of the trainee – it is the acid test

Measures would typically be business or organizational key performance indicators, such as: volumes, values, percentages, timescales, return on investment, and other quantifiable aspects of organizational performance, for instance; numbers of complaints, staff turnover, attrition, failures, wastage, non-compliance, quality ratings, achievement of standards and accreditations, growth, retention, etc.
The challenge is to identify which and how relate to the trainee’s input and influence. Therefore it is important to identify and agree accountability and relevance with the trainee at the start of the training, so they understand what is to be measured
▪ This process overlays normal good management practice – it simply needs linking to the training input
▪ For senior people particularly, annual appraisals and ongoing agreement of key business objectives are integral to measuring business results derived from training
4 Levels of Training Effectiveness

Example in Practice – CAPA

When building a training program, start with with the intended behaviors that will drive results. Evaluating our CAPA program, we have three key aims, which we can apply measures against.

BehaviorMeasure
Investigate to find root cause% recurring issues
Implement actions to eliminate root causePreventive to corrective action ratio

To support each of these top level measures we define a set of behavior indicators, such as cycle time, right the first time, etc. To support these, a review rubric is implemented.

Our four levels to measure training effectiveness will now look like this:

LevelMeasure
Level 1: Reaction Personal action plan and a happy sheet
Level 2: Learning Completion of Rubric on a sample event
Level 3: Behavior Continued performance and improvement against the Rubric and the key review behavior indicators
Level 4: Results Improvements in % of recurring issues and an increase in preventive to corrective actions

This is all about measuring the effectiveness of the transfer of behaviors.

Strong Signals of Transfer Expectations in the OrganizationSignals that Weaken Transfer Expectations in the Organization
Training participants are required to attend follow-up sesions and other transfer interventions.

What is indicates:
Individuals and teams are committed to the change and obtaining the intended benefits.
Attending the training is compulsory, but participating in follow-up sessions or oter transfer interventions is voluntary or even resisted by the organization.

What is indicates:
They key factor of a trainee is attendance, not behavior change.
The training description specifies transfer goals (e.g. “Trainee increases CAPA success by driving down recurrence of root cause”)

What is indicates:
The organization has a clear vision and expectation on what the training should accomplish.
The training description roughly outlines training goals (e.g. “Trainee improves their root cause analysis skills”)

What is indicates:
The organization only has a vague idea of what the training should accomplish.
Supervisors take time to support transfer (e.g. through pre- and post-training meetings). Transfer support is part of regular agendas.

What is indicates:
Transfer is considered important in the organization and supported by supervisors and managers, all the way to the top.
Supervisors do not invest in transfer support. Transfer support is not part of the supervisor role.

What is indicates:
Transfer is not considered very important in the organziaiton. Managers have more important things to do.
Each training ends with careful planning of individual transfer intentions.

What is indicates:
Defining transfer intentions is a central component of the training.
Transfer planning at the end of the training does not take place or only sporadically.

What is indicates:
Defining training intentions is not (or not an essential) part of the training.

Good training, and thus good and consistent transfer, builds that into the process. It is why I such a fan of utilizing a Rubric to drive consistent performance.

Management’s Job

In episode 48 of the Deming Len’s podcast, the host refers back to Deming’s last interview, “Dr. Deming: ‘Management Today Does Not Know What Its Job Is‘”

Deming Lens #51 – Pay for Performance The W. Edwards Deming Institute® Podcast

In our 51st Deming Lens episode, host Tripp Babbitt shares his interpretation of wide-ranging aspects and implications of Dr. Deming's System of Profound Knowledge. This month he looks at supply chain and risk management. Show Notes [00:00:14] Deming Lens – Episode 51: Pay for Performance [00:02:19] A Journey with Pay for Performance [00:03:28] The New Harmony Project [00:04:33] The Result of Collectivism [00:06:35] Analytical vs Synthetic Thinking [00:10:19] Dr. Deming and Pay for Performance [00:12:22] Pay for Performance, Collectivism and Dr. Deming     Transcript [00:00:14] In the 51st episode of the Demning lens, we'll talk about. Pay for performance. Hi, I'm Tripp Babbitt, host of the Deming Institute podcast, and this is the fifty first episode of the Deming Lens. And as I was going back and searching through some of the old episodes I've done, I noticed I hadn't done one on pay for performance. And it kind of came to the forefront of my mind when I was going through some Tic Tacs. And there's a couple of people out there that are big advocates of pay for performance as ways to keep people in their organizations, which is becoming a problem now. You know, people aren't going back to work for variety reasons. The stimming check certainly plays a role in that, working from home, people being called back to work when they had got used to being at home. There's this thing that goes around on LinkedIn about you only live once and you might as well do something that that makes you happy. And organizations these days don't seem to be making people very happy. And so these are some of the topics that I think I want to talk about with regards to pay for performance. And I actually put a whole YouTube video together on my personal YouTube channel responding to Tessa White, who goes by the name Job Doc Tessa. And she used to get some pretty good advice. But one of the pieces of advice was if you're going to keep people that you got to pay for performance.   [00:02:19] So let's talk about that, because it's been an interesting journey with regards to pay for performance and at least in my own personal journey, doing public and private seminars where we would talk about pay for performance, which was something that Dr Deming obviously advocated against. And I'd often get a lot of pushback on this. You know, when you're talking about getting rid of incentives or paying for performance at almost every public or private seminar that I did, I get some pushback either allowed during the course of the seminar or privately afterwards that asking if Dr Deming was a socialist or a communist. And I've done other episodes on this particular subject, but I'm from Indiana and in Indiana. Back in the early 1980s.   [00:03:28] There was a project called and New Harmony, Indiana, that basically advocated for collectivism, which was put together as a utopian society by a gentleman, by the British industrialists, by the name of Robert Owen from 1825 to 1827. So let's have a look, maybe a little bit over a year. And then it was dissolved in 1827 because this whole concept of social education and equality did not work well for a lot of different reasons. But the idea of a utopian society in New Harmony, Indiana, was to eliminate crime and poverty and increase health, decrease misery and increase your intelligence and happiness. And this was what this community of equality was supposed to achieve in Indiana.   [00:04:33] And but what happened over time is that people discovered that the equality that this society was advocating, that there were certain rare and very hard working people, you know, putting together the agriculture that was needed for the society or building houses or whatever, is that the hard working realize that they were earning the same as the laziest of the people in the group and that caused conflict. And then people stopped working. So food wasn't being grown and houses weren't. Their houses and buildings were being built because there was kind of a pointing fingers and blame and things of that sort. So this whole utopian society based on collectivism was didn't work out. It collapsed very quickly and. You know, when you when I'm talking about pay for performance in public and private seminars. You know, the thing I'm communicating to people is that the individual cannot be separated from the system that you work in. And what I discovered over the years is there's there's something called Dunbar's number, which is 150. When an organization grows to about 150, you you become it becomes more and more bureaucratic. And the larger it grows, the more bureaucratic that it becomes. And managing that becomes very difficult. So the larger your system is, the harder it is to separate the performance of an individual from the system that that you're working in. Now, I've seen it get really bad, anywhere from 75 to 150 employees and working with some smaller organizations.   [00:06:35] And but it's borne the pay for performance. The mindset is borne from what I referenced is analytical versus synthetic thinking. And that is that, you know, we break down the pieces in analytic and analytical thinking and we break them down all the way to the individual and say, OK, here's your goals. If you just meet your goals and everybody meets their goals, the company's going to do well. And I think if you go back to the interview I did with Paula Marshall, you'll discover that you can't separate the individual from that system and that the system is greater than the sum of its parts. So the whole concept of synthetic thinking or systems thinking gives you a different view of how your organization works. And it's not something that you can just break apart all the way down to the individual and add it up. It's exponential. And managing the interconnections in a system is where you get your largest gains. So, you know, it's been interesting from my standpoint that we have gone from this concept of Dr. Deming's ideas around pay for performance, where people were, in essence, questioning whether he was a socialist or a communist or whatever. To today we seem to have this pendulum swinging, swinging where, you know, everybody should be getting the same pay. And I you know, the things that Dr. Deming worked on from my studies and my using his system of profound knowledge are getting clarity of purpose.   [00:08:29] You know, if you if you understand your system and you understand your connection to the customer, that raises your level as a worker, understanding kind of what you do as opposed to just being kind of this cog in the system that just does its thing. And there's no benefit to doing that, but that people are paid differently based on the skills that they get. And developing those skills becomes something that the organization would want to do for its own benefit, as well as the benefit of the individual. One other thing Dr. Deming advocated is innovation and that people are participating and coming up with ideas and more and more. I've seen this huge conflict in organizations between the executives micromanaging things. And, you know, you do your work and you do it in this way, as opposed to giving more responsibility that the worker has even greater knowledge because they're actually doing the work to make their own decisions about their own work. And that management's job is to remove the barriers that create problems for the workers. And they're facilitating things, moving, moving forward. And the best organizations that I've worked with either working towards that or they're getting the knowledge to work towards that. And this is what will keep people there. Never will there be this collectivism type of mentality on the one and one side of things where, you know, everybody gets the same pay.   [00:10:19] And on the other end, where there's this pay for performance, there's Dr. Deming is proposing something very different than those two concepts to be able to make the organization better and have individuals want to be a part of that and not go to things like wanting to be an entrepreneur. Fighting, though, the work from home, you know, going to a smaller company or just staying waiting till the clock runs out on their stimming checks, these are all things that are preventing us from moving forward. Now, one thing I am getting more and more concerned about is, you know, we need these large organizations to achieve big things. You know, an individual entrepreneur has to be very fortunate. You know, there's very few Jeff Bezos of the world that start from scratch and then grow a particular company into a behemoth like Amazon. And so we need these large organizations and we need them to run well because they can achieve big things, whether it's making cars or homes or whatever it is. These are very large systems and people get lost in the shuffle in the way that things are happening today. And so they're dissatisfied. We have a whole generation of people that are really caught up and trying to do something for the greater good. But I think it gets a little bit misguided. And I believe that some of the organizations that we have today with the mindset of micromanaging people and control and incentives and pay for performance are all things that are making their systems worse.   [00:12:22] And, you know. We don't need pay for performance and we don't need collectivism, but what we do need are good systems that people want to work in on a daily basis. And so that's that's the message I wanted to communicate this month in the Deming lens is this concept of pay for performance where you can separate an individual from the system, which you can't. And the larger the system is, the harder that becomes. If you're an entrepreneur, then, yeah, pay for performance. And that's that's exactly what you get when you're an entrepreneur, you know, a company of one. But as it grows, there needs to be a better an organization built on a clarity of purpose, their connection to the customer, and that the person, no matter where they are in your organization, can understand that what the customer is looking for. And I use what's called a customer lens to be able to achieve that and innovation, the ability to share ideas that they have about the work and the way that it should be done and the ownership of that by the worker as opposed to management and executives and people of that sort. So that was it for this week. I'll talk to you again next month for the latest videos, blogs, events and information from the Dummying Institute. Be sure to go to Deming dot org.
  1. Deming Lens #51 – Pay for Performance
  2. Deming Len's Episode #49 – Supply Chain and Risk Management
  3. Deming Len's Episode #48 – Management (Still) Doesn't Know What Its Job Is
  4. Deming Len's Episode #47 – SoPK: The Interaction of the Parts
  5. Deming Lens #46 – The Art of Tampering

I’ve written recently about driving fear out of the organization. Without a doubt I think this is the number one task for us. True North for the quality profession.

The source of innovation is freedom. All we have—new knowledge, invention—comes from freedom. Somebody responsible only to himself has the heaviest responsibility. “You cannot plan to make a discovery,” Irving Langmuir said. Discoveries and new knowledge come from freedom. When somebody is responsible only to himself, [has] only himself to satisfy, then you’ll have invention, new thought, now product, new design, new ideas.

Dr. W. Edwards Deming

Whataboutism

Whataboutism is the common term for a version of the tu quoque fallacy, a diversionary tactic to shift the focus off of an issue and avoid having to directly address it by twisting criticism back onto the critic and in doing so revealing the original critic’s hypocrisy.

Whataboutism often results in a comparison of issues as pure deflection. We see it when individuals are always focused on why others get ahead and they don’t, looking for comparisons and reasons they are being treated unfairly instead of focusing on their own opportunities for improvement. It is so easy to use when we are faced with criticism, “Well, what about … ?”

We also see whataboutism in our cultures. Maybe it is a tendency to excuse your own team’s shortcomings because obviously the sins of another team is so much worse. This is a result of, and strengthens, silo-thinking.

Building the feedback process to reduce and eventually eliminate whataboutism is critical.

Transparency

Transparency in organizations generally falls into one of two categories: proactive and demand-driven.

  • Proactive dissemination refers to information that the organization’s leadership makes known about its activities and performance. Practical expressions can range from strategic and tactical goals, third-party evaluations, and organizational risks.
  • Demand-driven access refers to an organizational commitment to respond to employee requests for specific kinds of information or documents which otherwise would not be accessible.

The idea of transparency can also be unpacked in terms of its directionality. Disclosure cuts both ways, channeling information upwards as well as downwards in the organization. Leadership sharing insight into decisions is an example of downwards, where idea management processes are upwards.

The type of transparency also matters:

  • Opaque or fuzzy transparency involves the dissemination of information that does not reveal how organizations actually behave in practice, whether in terms of how they make decisions, or the results of their actions. The term also refers to information that is divulged only nominally, or which is revealed but turns out to be unreliable.
  • Clear transparency sheds light on organizational behavior, which permits interested employees to pursue strategies of constructive change.

This distinction between clear and opaque is grounded on the premise that if transparency practices are going to meet their goals of transforming organizational behavior, then they must be explicit in terms of who does what, and who gets what.

Transparency requires not only a willingness, but processes to drive it.

Photo by Elena Koycheva on Unsplash

Drive Out Fear on International Workers Day

Happy International Workers Day. Let’s celebrate by Driving Out Fear!

Thirty-five years ago Deming wrote that “no one can put in his best performance unless he feels secure.” Unfortunately, today we still live in a corporate world where fear and management by fear is ubiquitous. That fear is growing after more than a year of a global pandemic. As quality professionals we must deal with it at every opportunity.

Fear undermines quality, productivity, and innovation. The existence of fear leads to a vicious downward spiral.

Some sources of fear include:

  • Competition: Many managers use competition to instill fear. Competition is about winners and losers. Success cannot exist without failure. Managers deem the anxiety generated by competition between co-workers a good thing as they compete for scarce resources, power and status. Therefore, management encourage competition between individuals, between groups and departments and between business units.
  • “Us and Them” Culture: The “us and them” culture that predominates in so many organizations proliferated by silos. Includes barriers between staff and supervisors.
  • Blame Culture: Fear predominates in a blame culture. Blame culture can often center around enshrining the idea of human error.

We drive out fear by building a culture centered on employee well-being. This is based on seven factors.

FactorMeansObtained by
ResponsibilityWell defined responsibilities and ownershipThe opportunity an employee has to provide input into decision making in his department
An individual employees’ own readiness to set high personal standards
An individual employee’s interest in challenging work assignments
The opportunity an employee has to improve skills and capabilities
Excellent career advancement opportunities
The organization’s encouragement of problem-solving and innovative thinking
Management CompetenceManagers trained with skills that lend themselves to contributing to the work of their team ensures that they will be looked to for help. Managers need to be able to guide.Direct Supervisor/Manager Leadership Abilities Management is engaged and leads by example (Gemba walks)
Management by Facts
ConsiderationWhen managers act as if employees have no feelings and just expect them to do their work as if they are robots, it can make employees uneasy. Such behavior makes them feel detached and merely a tool to carry out an end. In such environments, many times the only times employees hear from the manager is when something goes well or really bad. In either case, the perception could be that the manager has mood swings and that also adds to the employee’s insecurity. They may feel reluctant to talk to their manager for fear he is in one of his bad moods.Senior Management’s sincere interest in employee well-being
An individual employee’s relationship with their supervisor
Open and effective communication
Trust in management and co-workers
CooperationThe feeling that every person is on their own to look out for their interest is a sad state to be in. Yet when everyone has a fear that the other workers will take advantage of them or make them look bad at the first opportunity, a selfish and insecure environment will result. Employees should be able to work together for the benefit of the company. They should focus on group goals in addition to their personal goals, recognizing that individually there will be failures, but that the whole is more important than the individual parts.Trust Well trained employees Collaboration as a process Organizational culture (psychological safety) Hire and promote the right behaviors & traits to match the culture
FeedbackInformation that is given back to the employee regarding their performance on the job.Know what is expected of them (clear job descriptions)
Effective processes for timely feedback
Recognition
Know their opinion matters
InformationTransparency is critical. When employees know nothing about how a company is doing in terms of where they should be, it is a source of uneasiness. Without that knowledge, for all they know the company could be doing very poorly and that could be a bad thing for everyone. When they have a better sense of where the company is in the scheme of their objectives set by management, it helps them feel more secure. That is not to say it is the news being good or bad that affects their security, but rather the fact that they actually have the news.Strategy and Mission — especially the freedom and autonomy to succeed and contribute to an organization’s success
Organizational Culture and Core/Shared
Values
Feel that their job is important
StabilityEmployees feel more secure when their role does not change frequently and they understand what tomorrow will mean.Job Content — the ability to do what I do best
Availability of Resources to Perform the Job Effectively
Career development – opportunities to learn and grow
Photo by fauxels on Pexels.com