Plan an evaluation across reaction, learning, behavior and results, starting from the business outcome.
Why it matters
"Did the training work?" is the question every sponsor eventually asks. Most training is evaluated only with an end-of-session satisfaction form, which tells you whether people liked it, not whether it changed anything. Evaluation models give you a structure for gathering better evidence, and planning evaluation before delivery (backward design again) makes it far easier.
Kirkpatrick's four levels
Donald Kirkpatrick introduced four levels of training evaluation in a series of articles in 1959. They remain the most widely used framework.
| Level | Question | Typical measures | Incident-response example |
|---|---|---|---|
| 1. Reaction | Did they find it engaging and relevant? | End-of-session survey | "How useful will this be on your next shift?" (1 to 5) |
| 2. Learning | Did they gain the knowledge, skills and confidence? | Tests, practical tasks, pre/post | Severity scenario test before and after; sandbox escalation task |
| 3. Behavior | Do they apply it on the job? | Observation, work samples, manager check-ins, system data | Severity accuracy on real incidents in the first 60 days |
| 4. Results | Did the targeted business outcomes improve? | Business metrics | Mean time to resolve customer-impacting incidents |
The New World Kirkpatrick Model
James and Wendy Kirkpatrick updated the model (Kirkpatrick's Four Levels of Training Evaluation, 2016). Key changes:
- Start planning at Level 4 and work backwards: what result does the business need, what behaviors drive it, what must people learn, and how should training feel?
- Required drivers at Level 3: processes and systems that reinforce, monitor and reward the new behavior on the job (manager coaching, job aids, follow-up). Without them, learning rarely turns into behavior.
- Leading indicators at Level 4: short-term signs that results are on track (for example, escalations reaching the right team first time).
- Level 1 adds engagement and relevance, and Level 2 adds confidence and commitment.
Phillips ROI: a fifth level
Jack Phillips added Level 5: Return on investment, comparing monetary benefits with program costs:
ROI (%) = (net program benefits / program costs) x 100
net program benefits = monetary benefits - program costs
Example: the onboarding program costs 40,000 (design time, trainer time, learners' time). Faster incident resolution is estimated to save 70,000 in the first year. Net benefits = 30,000, so ROI = 30,000 / 40,000 x 100 = 75%. The hard part is isolating the effect of training from everything else that changed (a new tool, a quieter quarter). Phillips recommends methods such as control groups, trend-line analysis and conservative estimates, and being explicit about assumptions.
Criticisms and alternatives
- The levels are not a chain. George Alliger and colleagues' 1997 meta-analysis found that reaction measures were only weakly related to learning and transfer. People can enjoy training and learn little, or dislike challenging training that works. Do not use Level 1 as a proxy for Level 2 or 3.
- Most effort stays at Level 1. Surveys are easy; behavior and results take planning and stakeholder support.
- Will Thalheimer's LTEM (Learning-Transfer Evaluation Model, 2018) offers eight tiers, separating attendance and activity from knowledge recall, decision-making competence, task competence, transfer and effects of transfer, to discourage treating weak measures as strong evidence.
- Robert Brinkerhoff's Success Case Method studies the most and least successful participants in depth to find out what helped or blocked transfer, which is often more persuasive to stakeholders than averages.
Planning a four-level evaluation
For the incident-response onboarding:
- Level 4 (start here): target is to reduce mean time to resolve SEV2 incidents handled by new engineers; leading indicator: correct first escalation rate.
- Level 3: severity accuracy on a weekly sample of real incidents for 60 days; required drivers: buddy reviews, matrix linked in the tool, manager check-in at day 30.
- Level 2: pre/post scenario test and the sandbox escalation task; confidence rating.
- Level 1: three-question survey (relevance, confidence, one improvement) at the end of week 1.
- Timing and owners: who collects what, when, and where the data lives.
- Baseline: collect the same measures for the previous cohort, or before launch.
Agree this plan with the sponsor before the program starts, including which results would count as success.
Common mistakes
- Only surveying satisfaction.
- Measuring behavior with no baseline.
- Claiming all improvement was caused by training.
- Ignoring the work environment that supports or blocks new behavior.
- Choosing measures after the program, based on what looks good.
Quick recap
- Kirkpatrick: reaction, learning, behavior, results.
- Plan from Level 4 backwards; build required drivers for Level 3.
- Phillips adds ROI; isolate the training's effect and state assumptions.
- Reaction does not predict learning; consider LTEM and success cases too.
Try it
Answer the quiz on the Do tab, then draft the Level 4 and Level 3 measures for a training you know.