skip navigation
Environmental Factor, November 2012

WETP promotes logic model concept for training assessment

By Eddy Ball

WETP Director Chip Hughes

Hughes could joke about Kirkpatrick, even as he described the WETP dilemma when it comes to evaluation. “We have one foot in the real world,” he observed, “and one foot in the academic world.” (Photo courtesy of Steve McCaw)

WETP grantee Craig Slatin, Sc.D.

Slatin lamented what he called a 40-year effort to reduce worker power that has resulted in the unemployment, or chronic underemployment, of some 23 million workers in the U.S. (Photo courtesy of Steve McCaw)

A workshop at NIEHS Oct. 11-12 aspired to fundamentally transform ways of thinking about worker health and safety training evaluation, as well as share best practices. More than 100 grantees and contractors attended the workshop, which was part of the annual meeting (http://tools.niehs.nih.gov/wetp/events.cfm?id=2521) of the NIEHS Worker Education and Training Program (WETP).

The workshop developed a theme that WETP Director Chip Hughes introduced in his opening comments on the importance of being able to demonstrate the effectiveness of worker health and safety training and its value to the nation. “Program evaluation has always been a core part of our mission.”

As veteran evaluator Ruth Ruttenberg said during a panel session on evaluation tools and methods, “Training only matters if it makes a difference.” Her challenge, and the challenge of the other panelists and presenters, involved helping participants think critically about how they design training and how they can develop metrics to show its impact on workers on the job, at home, and in the larger political context of their lives.

Following Hughes' introduction, WETP grantee Craig Slatin, Sc.D., presented a big-picture keynote address on “Current and Future Safety and Health Training Expectations Under 21st Century Workplace and Socioeconomic Conditions.” In the face of economic restructuring that continues to impact job security, Slatin explained, the objectives of training can be difficult to realize. “We try to get workers’ voices out through the training,” he said, “[but the question of] how far do I push is always in workers’ minds.”

Something old, something new — Kirkpatrick and logic models

Hughes' introduction, and nearly every presentation at the workshop, acknowledged the tremendous contribution and continuing relevance of the four-level model of training course evaluation first published by University of Wisconsin Professor Emeritus Donald Kirkpatrick, Ph.D., (http://www4.uwm.edu/sce/instructor.cfm?id=12884)  in 1959. Modified over the years by the author and his partners, Kirkpatrick’s framework (http://www.kirkpatrickpartners.com/)  breaks down evaluation into how well learners respond, what they learn specifically, how much training influences performance, and whether the education program accomplished its original organization goals.

Not surprisingly, Kirkpatrick’s influence is also evident in the WETP logic model, which was the topic of presentations by NIEHS Division of Extramural Research and Training program analyst and evaluation specialist Kristi Pettibone, Ph.D., WETP program analyst Jim Remington, and WETP senior intern Hannah Leker, as well as a hands-on group exercise involving a collective effort to flesh out a logic model for their own programs.

The logic model reflects Kirkpatrick’s striving for exhaustive analytical precision, as it helps users construct a timeline flow chart of inputs, activities, outputs, and impacts — short term, midterm, and long term. The WETP handout also included mission and organizational priorities from the WETP 2008-2113 strategic plan (http://www.niehs.nih.gov/careers/assets/docs/wetp_strategic_plan.pdf)  as a benchmark for planning activities.

Numbers, narratives, and post-training audits

Early in the workshop, panelist Sue Ann Sarpy, Ph.D., introduced the concept of 360-degree evaluation. She described the feedback evaluation scheme as a multisource system that includes all stakeholders, including the director, students, employers, program coordinator, and community members, and also serves as a continuing quality improvement instrument. Sarpy’s approaches range from questionnaires and open-end questions, to focus groups and success stories to help lend a human touch to other metrics, as well as strengthen the case for support.

The 360-degree approach can also involve audits of trainees after they’ve completed training, with a checklist of behaviors observed. Yale University project manager Thomas Ouimet warned against the number one fatal assumption — attending equates with learning — as he advocated for on-the-jobsite followup observations. Like many of his co-presenters, Ouimet is a strong advocate of blended training, such as e-learning simulation, role playing, and case studies, as well as blended assessment with instruments matched to Kirkpatrick’s levels of evaluation.

As he wrapped up the meeting, Hughes looked toward funding realities, and the aging of the current group of experienced health and safety program operators and trainers. The workshop title, “Prove It Makes a Difference,” referred both to the immediate, as well as long term, health of worker training. “The impact on the next generation is one of our big challenges for the future,” he said.


Ruth Ruttenberg

Looking back on the successes of worker training, Ruttenberg told participants, “We should have had an epidemic from cleaning up hazmat, but we didn’t.” She added that, today, there are first responders in every state in the U.S.(Photo courtesy of Steve McCaw)


Cathy Cronin

Occupational Safety and Health Administration training director Cathy Cronin called for active, job-focused training. She said instructors should devote about one-third of class time to lecture and two-thirds to interaction.(Photo courtesy of Steve McCaw)


Don Elisburg, J.D., of the National Clearinghouse for Worker Safety and Health Training

Elisburg is but one of several veterans of the program who are looking at retirement in the near future. Looking ahead, he encouraged trainers to keep in mind the very different learning styles of the next generation of workers.(Photo courtesy of Steve McCaw)


Kristi Pettibone, Ph.D.

An author of the NIEHS Partnerships for Environmental Public Health Evaluation Metrics Manual (see story), Pettibone helped lead the group activities that resulted in creation of the WETP logic model.(Photo courtesy of Steve McCaw)


Jamie Kirkley, Ph.D.

Jamie Kirkley, Ph.D., of Wisdom Tools Enterprises, underscored the importance of precise analysis of training design and desired outcomes. “Don’t teach them [trainees] to swim, if you end up testing them on diving,” she told the participants.(Photo courtesy of Steve McCaw)


Paul Schulte, Ph.D.

Schulte said that employers should look at worker safety and health training as one part of a comprehensive risk management program.(Photo courtesy of Steve McCaw)


Worker safety and health training — a 25-year odyssey

Although the workshop focused on evaluation as a defensive strategy in an atmosphere of flat and even declining budget resources, participants have a rich history of accomplishment to build on, as they make their case. “You have to prove to somebody what you already know is true,” Hughes said to the audience. He also reflected on the quality of what crowd-source knowledge trainers can bring to the effort.

Keynote rappateur Eula Bingham, Ph.D., of the University of Cincinnati, framed her talk on evaluation in historical terms. “Let people know that you’re saving lives,” she said, “[by] helping us to tell the people it was all worthwhile.”

National Institute for Occupational Safety and Health researcher Paul Schulte, Ph.D., added, “WETP is one of the unsung heroes of this country.” Looking ahead to the potential for taking an effective program, with a proven track record, to an even higher level, Don Elisburg, J.D., of the National Clearinghouse for Worker Safety and Health Training, talked optimistically about what the talent on display during the workshop could mean for the future of worker training. “Looking at what everybody has done with these projects is mind-blowing,” he said.



"Birnbaum gives keynote at ..." - previous story Previous story Next story next story - "Superfund marks 25th year ..."
November 2012 Cover Page

Back to top Back to top