Environmental Factor, February 2011, National Institute of Environmental Health Sciences
Measuring efforts to translate science into public health action
By Eddy Ball
NIEHS DERT Director Gwen Collman, Ph.D., welcomed workshop participants. "It became apparent years ago that there weren't very many metrics related to evaluation that were readily available," she told the audience. (Photo courtesy of Steve McCaw)
Drew explained that NIEHS is not trying to prescribe program evaluation procedures. "What we are trying to do with this manual is to show you how you can come to those metrics yourself," she said. (Photo courtesy of Steve McCaw)
The workshop attracted science education outreach specialists, including, from foreground, Bono Sen, Ph.D., Environmental Health Perspectives (EHP) science education outreach program manager; Carly Carroll, EHP science education program coordinator; Kelly Leovic, Environmental Protection Agency environmental education program manager; and Ericka Reid, Ph.D., NIEHS education outreach specialist. (Photo courtesy of Steve McCaw)
Members of the evaluation metrics team were on hand to help field questions about the manual. Shown above are Program Analyst Helena Davis, left, and Health Scientist Administrator Kristi Pettibone. (Photo courtesy of Steve McCaw)
Social Research Specialist Alison Gunn, representing the Clinical and Translational Science Awards program at the University of North Carolina at Chapel Hill, was at the workshop to learn more about the new manual. Seated to her right is colleague Zoe Enga, evaluation and data coordinator. (Photo courtesy of Steve McCaw)
Program Analyst Liam O'Fallon oversees NIEHS environmental justice grants. He told the audience that a better understanding of metrics would help NIEHS staff as well benefit grantees. (Photo courtesy of Steve McCaw)
NIEHS took a step toward helping grantees measure the progress of outreach and translation goals at a workshop Jan. 10, with the formal launch of its new Evaluation Metrics Manual(http://www.niehs.nih.gov/research/supported/programs/peph/materials/index.cfm). The manual targets programs in the NIEHS Partnerships for Environmental Public Health (PEPH)(http://www.niehs.nih.gov/research/supported/programs/peph/index.cfm) program (see text box) with guidance for researchers and program staff on ways to use logic models to define what constitutes success and how to measure it.
Led by the manual's senior author, NIEHS Program Analysis Branch Chief Christie Drew, Ph.D., the workshop was the first in a series of training and information-gathering sessions expected to continue through February.
"Part of the challenge is that many products that are developed by PEPH outreach and translation activities do not make it into the typical peer-reviewed literature, which is how NIH often measures the progress of knowledge creation," Drew said. "So we need to think about what other kinds of approaches we can use to document grantee success."
A work in progress
Drew emphasized that the draft is not a prescriptive design for planning and evaluation, but a collection of tangible examples from effective programs representing a starting point for what should be seen as a dynamic, living document. She encouraged public comments on the draft through an evaluation metrics feedback form or by e-mail(mailto:email@example.com), and offered to schedule webinars for interested groups. She described the process as an ongoing partnership effort by NIEHS staff and grantees from a range of programs funded by the NIEHS Division of Extramural Research and Training (DERT).
The effort to develop evaluation metrics, Drew explained, addresses public responses to a 2007 "Request for Information: Partnerships for Environmental Public Health"(http://www.niehs.nih.gov/research/supported/programs/peph/about/rfi.cfm) and participant comments during a subsequent PEPH workshop(http://www.niehs.nih.gov/research/supported/programs/peph/about/workshop.cfm) held in June 2008.
PEPH programs and similar efforts funded by a variety of other organizations typically form partnerships with communities and stakeholders, to conduct and translate environmental health science research into new policies and programs to improve public health. While the goals may be clear - a policy decision by a government body or the removal of a potential health threat from a community, for example - in many cases, the specific steps toward those goals can be harder to define and even more challenging to measure.
A logic model for evaluation
The manual employs a logic model framework to analyze how programs can potentially use various resources or inputs to conduct a range of activities that generate a series of products or outputs to realize benefits or impacts. It focuses on five key program activity areas - partnerships, leveraging, products and dissemination, education and training, and capacity building for communities, researchers, healthcare professionals, and decision makers.
The manual's appendices include a list of NIEHS staff and subject expert discussants and discussant dates, additional evaluation resources, a combined bibliography of the more than 150 references cited in the manual, and a cross index of examples and chapter sections.
To review the manual and provide comments, visit the PEPH website(http://www.niehs.nih.gov/research/supported/programs/peph/materials/index.cfm).
Partnerships for Environmental Public Health (PEPH) - an umbrella program at NIEHS
With its roots in environmental justice and community-based participatory research programs, PEPH brings together scientists, community members, educators, health care providers, public health officials, and policy makers in the shared goal of advancing the impact of environmental public health research at local, regional, and national levels. The program emphasizes both scientific advances and the development of practical materials for use in communities, with a focus on translating research findings into tools, materials, and resources that can be used by a variety of audiences to prevent, reduce, or eliminate adverse health outcomes caused by environmental exposures.
Members of PEPH conduct programs funded by a broad range of funding mechanisms, including center grants for the promotion of children's and women's environmental health, outreach and translation components of the Superfund Research Program, and education and training funded by the Worker Education and Training Program. The program's cross-divisional emphasis facilitates communication among partners who might otherwise have few opportunities to share their successes and challenges with one another.