Instructions for Use
The success of a Kirkpatrick Level 3 evaluation largely depends on the design of the evaluation. Evaluation approaches, including assessment design, data collection source and methods, measurement indicators, and data analysis and reporting, should be considered when developing a Level 3 evaluation strategy. Drivers of Level 3 evaluation strategies may include outcomes of interest, resources, cost, business needs, and time.
The Preparedness and Emergency Response Learning Centers (PERLC) Level 3 Working Group developed a table of possible evaluation approaches for assessing behavior change. It is provided to guide the evaluator in developing a Kirkpatrick Level 3 evaluation strategy. Approaches consider the design and data collection source and method of the Level 3 evaluation. Approaches are applicable across the continuum of field settings (i.e., everyday workplace, exercise or simulation, and real-life event) unless otherwise noted. Additionally, the matrix provides examples of types and sources of Level 3 measurement indicators. Finally, example questions for select approaches are provided.
Example Kirkpatrick Level 3 Questions: (back to top)Example 1: Questions for a post-test survey administered to participants of an exercise or simulation - Have you taken any training in the last 3-6 months that helped you perform your roles and responsibilities during this [setting] drill/exercise?
- If yes, briefly describe the purpose of the training or list the name of the training.
- Were there any lessons learned from the drill/exercise related to your roles and responsibilities? If so, please describe briefly.
- Could you perform all of your duties and tasks for which you were responsible?
- Consider including knowledge, skills, and attitudes (KSAs) from the training using a checklist during an annual review (this may be too confidential to share).
List all KSAs and have columns for ‘Yes’, ‘No’, or ‘Not Applicable’.
Example 2: Questions for behavior assessment that could be asked in the everyday workplace setting
(This basically asks respondents to think back to their behavior or the KSAs before the trainings and then to compare them after the trainings.)
- Have you taken any trainings in the last 3 to 6 months? (Yes/No) If so, please briefly describe the training or list the name of the training.
- Have you participated in any drills/exercises? (Yes/No) If so, did you perform your role and responsibilities accordingly?
- Did the trainings you have completed apply to your role and responsibilities?
- Thinking back prior to the trainings, would you have performed your role and responsibilities the same or differently? If so, please briefly explain.
- Were there any lessons learned from the drill/exercise? If so, please describe briefly.
- Could you perform all of your responsibility and task that you were given?
Example 3: Questions for behavior assessment that could be asked at different points in time in an everyday workplace setting (daily role) and/or exercise setting (exercise role)
0 no knowledge;
1 not mastered;
2 requires more knowledge or supervision;
3 requires little supervision; and
4 mastery level.
This could be done before the training, immediately after the training, and then again three to six months post training.
Back | Next