Outcome evaluation focuses on the effect the training had on learners and their organizations. Determining the results achieved through training and whether the desired effects were obtained are elements of outcome evaluation.
Strategies and Methods
- Plan for evaluation
- Base evaluation on program objectives
- Determine level(s) at which to evaluate
- Levels of evaluation (Kirkpatrick/Phillips) to consider:
- Level 1: Reaction
- Degree to which learners reacted favorably to the training
- Level 2: Learning
- Degree to which learners acquired the intended knowledge, skills, attitudes, and confidence
- Level 3: Behavior/Application
- Degree to which learners applied their new knowledge and skills on the job
- Level 4: Results/Organizational Impact
- Degree to which learners’ application of their new knowledge and skills produced the desired outcomes and impact
- Level 5: Return on Investment (ROI)
- Degree to which the benefits of the training (internal and external) outweighed the costs
- Consider available resources, time, data, expertise, and appropriateness of evaluating at a given level
- Identify key measures and indicators to evaluate based on program objectives
- Choose data analysis technique(s) appropriate to the evaluation level and target audience for evaluation (see Evaluation – Reporting Evaluation Results)
- Assign individual responsibilities for evaluation
- Establish evaluation team, if appropriate
- Collect baseline data based on evaluation level(s) chosen
- Collect data at the level(s) of evaluation selected
- Choose appropriate methods for data collection based on the level at which training is being evaluated
- Learners’ course ratings/evaluations
- Pre-tests/post-tests
- Questionnaires
- Tests/quizzes
- Exercises/case studies
- Demonstrations
- Observation checklists
- Simulations/drills/tabletop exercises
- Final/capstone projects
- Evaluation of work products
- Documentation of improved processes
- Customer satisfaction surveys
- Interviews/focus groups
- Choose appropriate timing for data collection based on the level at which training is being evaluated
- During training
- After training
- Choose appropriate sources for data collection based on the level at which training is being evaluated
- Learners
- Learners’ supervisors
- Learners’ subordinates
- Learners’ colleagues
- Learners’ organizations
- Clients/customers
- Trainers/facilitators
- Sponsors of training
- Analyze data collected
- Levels 1 (Reaction) and 2 (Learning)
- Descriptive statistics – sums, averages, rankings
- Comments from learners, trainers, sponsors
- Level 3 (Behavior/Application)
- Descriptive statistics – sums, averages, rankings, changes over time
- Performance reviews
- Comments from learners, trainers, sponsors, supervisors, subordinates, colleagues, clients/customers, organizations
- Level 4 (Results/Organizational Impact)
- Isolate the effects of the training on the desired outcomes
- Determine non-monetary benefits (and/or costs) that can be quantified
- Clients served
- Customer satisfaction
- Employee retention
- Determine monetary benefits (and/or costs)
- Reduction in costs, such as equipment and personnel
- Increase in revenue, such as from health insurers and grants
- Determine intangible benefits (and/or costs)
- Employee morale
- Reputation of organization
- Level 5 (Return on Investment)
- Complete Level 4 evaluation
- Convert data/impact measures (Level 4) to monetary value
- Calculate fully-loaded training costs
- Assessment costs
- Design and development costs
- Delivery costs
- Learner time
- Facilitator time
- Travel and lodging
- Facilities
- Materials
- Coordination
- Evaluation costs
- Calculate return on investment (ROI)
- Review outcome evaluation results
- Determine which desired outcomes were achieved
- Identify areas for improvement
- Suggest actions that can be taken to better achieve desired outcomes
Additional Resources