Assessment of long-term educational outcomes: Insights from the Kirkpatrick’s four level model of training criteria. Ludmila N. Praslova Vanguard University of Southern California Accrediting organizations, governments, and employers increasingly emphasize the long-term outcomes of education and expect institutions of Higher Education to prepare students for the labor force through development of relevant competencies, while also preparing them for civil participation (Geary Schneider, 2012; Toutkoushian, 2005; Voorhees and Harvey 2005). Documenting student achievement at the individual and societal levels is a complex task. Reactions are easily measured with indirect methods of assessment, but are considered insufficient for evaluation of outcomes (Kirkpatrick, 1959, 1976, 1996). Institutions are making progress in assessing learning with direct methods. Behavior and especially results criteria, while extremely important, are very difficult to evaluate and conceptualize. While organizations focus on organizational outcomes, current evaluation on Higher Education effectiveness is often focused on both societal and The classic model used for evaluation of training in individual outcomes (e.g., individual Return on business organizations, the Kirkpatrick’s four level Investment (ROI) (Gallup/Purdue, 2014; Chronicle of model of training criteria (Kirkpatrick, 1959, 1976, Higher Education, 2015). 1996), might provide a versatile framework for assessment in Higher Education (Praslova, 2010). Comparison between evaluation of training in organizations Higher Education suggests several The four levels of evaluation in Kirkpatrick’s model important questions to consider in evaluation of are reaction criteria, learning criteria, behavior results-level criteria: criteria, and results criteria. 1) Criteria In organizations In Higher Education Reaction (internal to program) trainee affective reactions & utility judgments student affective reactions & utility judgments Indirect assessment/student opinion surveys Learning (internal to program) direct measures of learning outcomes, typically knowledge tests or performance tasks direct assessment of learning outcomes (tests, papers, performance tasks, other graded work, national tests) Behavior (external to program) measures of actual on-the-job performance: supervisor ratings or objective indicators of performance/job outputs. Beyond course or program level. Evidence of student use of knowledge and skills learned early in the program in subsequent work, e.g., research projects or creative productions, application of learning during internship, through community involvement, and in the workplace. Other behaviors outside the context in which the initial learning occurred. Related documentation. Results (external to program) productivity gains, increased customer satisfaction, employee morale for management training, profit value gained by organization Table 1; in part based on Praslova, 2010. Individual level/Societal level: alumni career success, graduate school admission, service to society, personal stability, fulfillment etc. Alumni surveys, employer feedback, samples of scholarly or artistic accomplishments, notices of awards, recognition of service, etc. [email protected] 2) 3) 4) 5) 6) Are individual level vs. group-level results the appropriate level of analysis? If both, then what is the appropriate model to account for interactions between these? How does behavior translate into results, and how learning process can best influence behavior? Do measures such as jobs, ROI, and earnings tap into results level as intended by the proponents? Is national level productivity an appropriate outcome to assess? How much variance in individual outcomes is explained by education, individual characteristics, individual behavior, and contextual influences? How much variance in societal outcomes is explained by education vs. contextual influences of the national political and economic model and the global trends? Careful consideration of these questions should benefit attempts to evaluate the effectiveness of Higher Education. References: Gallup/Purdue (2014). Great Jobs, Great Lives. The 2014 Gallup-Purdue Index Report. Geary Schneider, C. (2012). Moving Democratic Learning from the Margins to the Core. Remarks Delivered at the White House. http://www.aacu.org/press_room/cgs_perspectives/documents/MovingDemocraticLearningfromtheMarginstotheCore.pdf Kirkpatrick, D. L. (1959). Techniques for evaluating training programs. Journal of the American Society of Training Directors, 13, 3– 9. Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: A guide to human resource development (2nd ed., pp. 301–319). New York: McGraw-Hill. Kirkpatrick, D. L. (1996). Invited reaction: reaction to Holton article. Human Resource Development Quarterly, 7, 23–25. Praslova, L. (2010). Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in Higher Education. Educational Assessment, Evaluation and Accountability, 22, 215-125, Springer Netherlands. DOI: 10.1007/s11092-010-9098-7 Toutkoushian, R. K. (2005). What can institutional research do to help colleges meet the workforce needs of states and nations? Research in Higher Education, 46, 955–984. doi:10.1007/s11162-005-6935-5. Voorhees, R. A., & Harvey, L. (2005). Higher education and workforce development: a strategic role for institutional research. New Directions for Institutional Research, 128, 5–12.
© Copyright 2018