Kirkpatrick evaluation

Why it matters

"Did the training work?" is the question every sponsor eventually asks. Most training is evaluated only with an end-of-session satisfaction form, which tells you whether people liked it, not whether it changed anything. Evaluation models give you a structure for gathering better evidence, and planning evaluation before delivery (backward design again) makes it far easier.

Kirkpatrick's four levels

Donald Kirkpatrick introduced four levels of training evaluation in a series of articles in 1959. They remain the most widely used framework.

The New World Kirkpatrick Model

James and Wendy Kirkpatrick updated the model (Kirkpatrick's Four Levels of Training Evaluation, 2016). Key changes:

• Start planning at Level 4 and work backwards: what result does the business need, what behaviors drive it, what must people learn, and how should training feel?
• Required driversProcesses and systems that reinforce, monitor and reward new behavior on the job, such as manager coaching and job aids. at Level 3: processes and systems that reinforce, monitor and reward the new behavior on the job (manager coaching, job aids, follow-up). Without them, learning rarely turns into behavior.
• Leading indicatorsShort-term signs that the targeted business results are on track, measured before the final outcome is known. at Level 4: short-term signs that results are on track (for example, escalations reaching the right team first time).
• Level 1 adds engagement and relevance, and Level 2 adds confidence and commitment.

Phillips ROI: a fifth level

Jack Phillips added Level 5: Return on investmentPhillips' Level 5: net program benefits divided by program costs, multiplied by 100 to give a percentage., comparing monetary benefits with program costs:

ROI (%) = (net program benefits / program costs) x 100
net program benefits = monetary benefits - program costs

Example: the onboarding program costs 40,000 (design time, trainer time, learners' time). Faster incident resolution is estimated to save 70,000 in the first year. Net benefits = 30,000, so ROI = 30,000 / 40,000 x 100 = 75%. The hard part is isolating the effect of training from everything else that changed (a new tool, a quieter quarter). Phillips recommends methods such as control groups, trend-line analysis and conservative estimates, and being explicit about assumptions.

Criticisms and alternatives

• The levels are not a chain. George Alliger and colleagues' 1997 meta-analysis found that reaction measures were only weakly related to learning and transfer. People can enjoy training and learn little, or dislike challenging training that works. Do not use Level 1 as a proxy for Level 2 or 3.
• Most effort stays at Level 1. Surveys are easy; behavior and results take planning and stakeholder support.
• Will Thalheimer's LTEMThalheimer's Learning-Transfer Evaluation Model, with eight tiers that separate weak measures like attendance from transfer and its effects. (Learning-Transfer Evaluation Model, 2018) offers eight tiers, separating attendance and activity from knowledge recall, decision-making competence, task competence, transfer and effects of transfer, to discourage treating weak measures as strong evidence.
• Robert Brinkerhoff's Success Case Method studies the most and least successful participants in depth to find out what helped or blocked transfer, which is often more persuasive to stakeholders than averages.

Planning a four-level evaluation

For the incident-response onboarding:

1. Level 4 (start here): target is to reduce mean time to resolve SEV2 incidents handled by new engineers; leading indicator: correct first escalation rate.
2. Level 3: severity accuracy on a weekly sample of real incidents for 60 days; required drivers: buddy reviews, matrix linked in the tool, manager check-in at day 30.
3. Level 2: pre/post scenario test and the sandbox escalation task; confidence rating.
4. Level 1: three-question survey (relevance, confidence, one improvement) at the end of week 1.
5. Timing and owners: who collects what, when, and where the data lives.
6. Baseline: collect the same measures for the previous cohort, or before launch.

Agree this plan with the sponsor before the program starts, including which results would count as success.

Common mistakes

• Only surveying satisfaction.
• Measuring behavior with no baseline.
• Claiming all improvement was caused by training.
• Ignoring the work environment that supports or blocks new behavior.
• Choosing measures after the program, based on what looks good.

Quick recap

• Kirkpatrick: reaction, learning, behavior, results.
• Plan from Level 4 backwards; build required drivers for Level 3.
• Phillips adds ROI; isolate the training's effect and state assumptions.
• Reaction does not predict learning; consider LTEM and success cases too.

Try it

Answer the quiz on the Do tab, then draft the Level 4 and Level 3 measures for a training you know.