Guiding Principles for Level 3 Evaluations​

Kirkpatrick Level 3 evaluations provide an opportunity to strengthen the chain of evidence between learning and changes to the performance of the individual. Likewise, evidence from Level 3 evaluations can provide insight as to factors that may be limiting the effectiveness of training programs when there is no indication of behavior change. To ensure learning transfers to behavior change and effectively measure the impact of learning on job performance, the following guiding principles and considerations are provided. 

1. Trainings vary in intensity and duration. Not all trainings are designed to produce behavior change (e.g., focused on awareness). The degree of behavior change is likely to be associated with the intensity and duration of the training. Some trainees are better able to use the material than others.
2. Evaluating behavior change as part of a training program is encouraged. Training programs consist of multiple sessions over time where the same objective is repeatedly targeted. For example, a competency may be targeted over several training courses. Repeated targeting of a competency over time is more likely to lead to behavior change.
3. There should be an alignment between the training and field setting objectives. To evaluate behavior change, the training and exercise objectives should be the same. For example, the capabilities required in a real world event should align with those taught and practiced during training.
4. To evaluate individual behavior changes over time requires that the same person be followed from the training environment to the field setting. 
• A protocol should be developed in advance. 
• Securing agreement of individuals for future cooperation is an important step to being able to measure longer-term behavior change. Permission (informed consent) and institutional review board (IRB) review are dependent on distinctions between quality improvement and research.
5. There must be sufficient resources and continuity to follow individuals from the training environment to the field setting.  Tracking individuals over time presents challenges. Characteristics of the re-evaluated sample should be compared with those “lost to follow-up." Sample differences will weaken conclusions that can be drawn.
6. Careful consideration should be given to what measures are used to assess behavior over time.
• Measures to assess individual achievement of particular competencies should align with the training objectives.
• Specific metrics should be valid and reliable, and have sufficient sensitivity to detect changes in behavior prior to and following training. 
• Strong quantitative measures are preferred for determining desired changes (effect sizes). Barriers to using quantitative measures should be assessed in advance.
• Including qualitative approaches can help to augment quantitative measures and enrich interpretation of findings. 
7. Other protocol considerations and caveats include:
• If a large proportion of the trainees do poorly at the end of training, there are several possibilities to be considered before longer-term follow-up is undertaken (i.e., trainees should be able to demonstrate a certain level of competence prior to expending resources on longer term follow-up).
• The trainees may not have been at an appropriate level to achieve the particular competency or competencies.
• There may have been shortcomings in how the training was marketed that may have attracted trainees who were not at an appropriate level to achieve the particular competency or competencies.
• The measure(s) used may not be reliable or valid and may need to be changed.
• The training itself may need to be adjusted to achieve the intended behavioral outcome(s).  
• If a real event (e.g., a tornado) occurs and the same trainees who did poorly function optimally, it will be important to ascertain whether trainees can attribute this to the specific training(s) or not.
• Keeping the Centers for Disease Control and Prevention's Six Steps and Relevant Standards for Program Evaluation in Public Health in mind when approaching any level of program evaluation is useful when assessing when to move forward and which type of approach might be useful (Turnock, 2009, p. 392).
  

Back  |  Next