Skip Navigation

Your Environment. Your Health.

Metrics Training

  • Introduction

    Partnerships for Environmental Public Health
    Evaluation Metrics Manual

    PEPH evaluation metrics manual cover
    This training will help you maximize use of the Evaluation Metrics Manual.
    Instructions
    • play arrow

      Click the arrow in black bar (above) to hear
      audio for each slide.

    • scroll example

      Use the Scroll Right/Left buttons or the navigation bar (above) to navigate through the training. This will advance the training one slide at a time.

    • The single arrow will skip 10 slides at a time.

    • The double arrows will jump to the beginning and end of the slide show.

    • Use the color coded navigation and progress bar to help you keep track of where you are:

      Screenshot of slider nav, for instructional purposes

    Last Updated: 4/4/2014

  • Welcome

    A welcome from Dr. Birnbaum

  • Introduction

    Click on the link below to access the Evaluation Metrics Manual.

    http://www.niehs.nih.gov/pephmetrics
     
    This training was developed by NIEHS as a companion to the PEPH Evaluation Metrics Manual.
  • Introduction

    Training Evaluation Form (22KB)

    Complete questions 1 and 2 before you start the training.
  • Introduction

    Purpose of Training

    PEPH Manual
    • Introduce Evaluation Metrics Manual
    • Develop a logic model
    • Identify appropriate metrics
    • Demonstrate the utility of Manual tools
    This training is intended to provide hands-on practice using the Evaluation Metrics Manual tools. It is not intended as a full evaluation course.
  • Introduction

    Partnerships for Environmental Public Health (PEPH)


    • UCLA Goods Movement open session
    • UTMB Network
    • UCLA Goods Movement
    • To find out more about the PEPH program: http://www.niehs.nih.gov/peph .
    • The PEPH network promotes interaction among grantees doing community-based work in the areas of Partnerships, Communication and Capacity Building.
  • Introduction

    Challenge


    • Idle Free Zone
    • University of Cincinnati Anti-Idling campaign
    • Paying With Our Health cover
    • Much of the great work in the PEPH program is not published in peer reviewed literature. It emerges as fact sheets, websites, curricula, and at town hall meetings, health fairs, etc.
    • Grantees asked for evaluation tools and approaches that were more relevant for this work.
  • Introduction

    Why Evaluate?

    • Design better projects
    • Improve outcomes
    • Facilitate a continuous improvement loop
    • Build stronger partnerships
    • Be able to replicate programs
    • Obtain additional funding
  • Introduction

    Evaluation Metrics Manual Purpose

    • Demonstrate how a systematic analysis can lead to meaningful evaluation metrics
    • Establish a common language for evaluation
    • Make evaluation more accessible
    • Build stronger partnerships
    • Intended audiences are:
      • - PEPH grantees and program staff
      • - Groups interested in measuring environmental public health programs
  • Introduction

    Program Areas Addressed in the Manual


    • PEPH Manual
    • PEPH Manual
    • PEPH Manual
    • Partnerships
    • Leveraging
    • Products and Dissemination
    • Education and Training
    • Capacity Building
  • Introduction

    The goal of the manual is to help you identify ways to measure progress and achievements in these areas using logic models.
  • Logic Models

    Logic models are a useful way to work with partners to identify key components of your program, including specific impacts you want to achieve and activities that you want to conduct in order to get to those impacts.
  • Logic Models

    Logic models are like a program road map.
    They help you figure out where you want to go, and how you are going to get there.
  • Logic Models

    Components of a Logic Model


    Components of a Logic Model
    • Inputs – resources available
    • Activities – actions that use available resources
    • Outputs – direct products of activities
    • Impacts – benefits or changes resulting from activities, outputs
  • Logic Models

    Linking Activities and Outputs

    Activities Outputs
    • If your activity is to create a message, then your output is that message.
    • If your activity is to conduct research that helps you understand behaviors around idling, dropping kids off etc., then your outputs might include a summary report of your findings.
    • If your activity is to educate bus drivers, one of your products might be a sticker for bus dashboards that remind drivers to turn their engines off.
  • Logic Models

    Video Clips of Activities and Outputs Discussion

    Watch this video for suggestions on how to identify activities and connect them to outputs.
  • Logic Models

    Logic models usually include inputs, context, and an indication of the relationships between the components.

    Components of a Logic Model
  • Logic Models

    Every aspect of a logic model is measurable, but you'll want to focus on those areas that are important to you.

    Components of a Logic Model

  • Logic Models

    We have shown you one way to draw a logic model, but there are lots of ways to illustrate your program.
    For more information on logic models, check out the Kellogg Foundation's Logic Model Development Guide 

    • Logic Model
    • Logic Model
    • Logic Model
  • Logic Models

    The manual's logic models are linear, and tend to be drawn to show increasing maturity from left to right and top to bottom.

    Activity Output Impact
  • Case Studies

    Click the links below to access PEPH case studies that include logic models and metrics.
    This video sets up the Cincinnati Anti-Idling Case study and outlines the exercise conducted in the next few slides.
  • Case Studies

    Logic Model Template

    • Try this exercise on your own.
    • Open up a logic model template (142KB) that you can use to illustrate key components of your specific program.
    • Take about 10 minutes to think about your program.
    • We encourage you to start at the end, and work your way back – so that you think about the impacts you want to achieve first. And then fill in the activities you'll conduct or outputs you might produce to get to those impacts.
  • Case Studies

    Let's Review Your Logic Model

    • Did you think about all the resources available to your project?
      • - Funding
      • - Staff
      • - Volunteers
      • - Equipment
      • - Partners
    • How about your activities? Did you use verbs to describe your activities?
      • - Conduct, Train, Identify, Create, Develop, Build
    • Are your outputs concrete, tangible products?
      • - Agendas, Curriculums, Maps, Databases
    • Do your impacts indicate a change in behavior, knowledge, or community or environmental factors?
  • Case Studies

    Logic Model Discussions

    Discussion of Goal 4
    Discussion of Goal 5
    Discussion of Goal 5 (Another Perspective)
    Difference between Goals and Impacts
    Outputs and Impacts Related to Changes in Knowledge
    Scoping your Work
  • Case Studies

    Key Points for Logic Modeling

    • Actively involve stakeholders. Different people bring different perspectives and expectations.
    • Write it down together.
    • Review to see progress and make changes.
    • Use to understand how goals connect to activities and impacts.
    • Allow you to document how the projects functions and what changes you want to achieve.
  • Metrics

    How Do We Get to Metrics?

    • A logic model provides a framework to clarify what is important
    • Measure what is important to you and your partners (these are your metrics!)
  • Metrics

    Measures of a characteristic or aspect of the program

    Ruler
    • For example: size, capacity, description, quality, quantity, duration, frequency
    • Qualitative or quantitative
    • Reportable and systematic descriptions of desired/actual performance or achievement
    • It is more difficult to measure a "partnership" than "length"
  • Metrics

    Strategies for developing metrics

    • Build metrics from the nouns, verbs, adjectives and adverbs that you use to talk about the program
    • Helpful questions include:
      • - What does that look like in practice?
      • - How would you recognize this when we see it?
  • Metrics

    Metrics Worksheet

    The   Metrics Worksheet (14KB) allows you access a worksheet that will walk you through some key questions you can ask yourself to figure out what is important to you and your project.

    Working through these questions will help you identify those things you may want to evaluate.
  • Metrics

    Check out the manual (11MB) for hundreds of sample metrics.

    Sample Metrics
  • Metrics

    Sample Activity Metrics

    Our activity example is:
    Build knowledge and skills
    Example activity metrics include:
    • The number of classes, workshops and other training sessions offered or attended
    • Descriptions of new skills obtained
    • Results of pre- and post-test questionnaires measuring changes in knowledge and skills
    • The number of forums where community members and health professionals meet to discuss environmental public health concerns (sponsored by PEPH partners)
    • The number of decision-makers who attend environmental public health seminars and workshops
  • Metrics

    Sample Output Metrics

    Our output example is:
    Community involvement in research
    Example output metrics include:
    • The number of partners who participate in collecting and analyzing data
    • Descriptions of community involvement in research process (what kinds of data collection and data analysis did they participate in?)
    • The number of partners who co-author papers
    • The number and descriptions of partners participating in seminars on campus and in the community, including numbers of contact hours
  • Metrics

    Sample Impact Metrics

    Our impact example is:
    Increased awareness of issues and research process
    Example impact metrics include:
    • The number of partners who report an increased awareness of environmental public health issues
    • Descriptions of how partners have applied their knowledge of the research process
    • The number of and descriptions of new programs that have been added to address research findings
    • Descriptions of how the community has mobilized around environmental public health issues
  • Metrics

    Access case study materials:


    • swing set
    • Idle Free Zone
    • boat
    The last few pages of every chapter also include a summary of the metrics discussed in that chapter.
  • Metrics

    How do you know if your metric is any good?
  • Metrics

    SMART metrics include the following 5 criteria that you can use to develop your metrics....
    • Specific
    • Measurable
    • Attainable
    • Relevant
    • Timely
  • Metrics

    Specific - detail the milestones you expect to achieve, who will achieve them and how
  • Metrics

    Measurable - define exactly what level of change you expect to achieve
  • Metrics

    Attainable - create a metric that your group or organization can actually achieve
  • Metrics

    Relevant - ensure your metric is connected to your goal
  • Metrics

    Timely - limit your metrics to those measures that you can reasonably collect within the time frame of the project
  • Metrics

    Data Collection

    Environmental Justice rally
    • Think about data collection and analyses early and often
    • Plan, but be open to new ideas
    • Include qualitative and quantitative data
    • Consider
      • - How many people will be needed?
      • - Will special equipment be required?
      • - How much training will be involved?
      • - How will data be stored?
      • - Who will own the data?
  • Metrics

    Exit interviews with training participants

    Revisiting a logic model
    Developing a logic model by working backwards
    Using a logic model to keep project on track
  • Metrics

    Discussions of Metrics for Data Collection Strategies

    During our metrics manual training, some of our participants had a great discussion about metrics that can be used to assess data collection strategies. The characteristics of the data that would be needed in the University of Cincinnati Anti-Idling case study include:
    • Appropriate capacity at the school to monitor and/or collect data
    • Asthma specific health data
    • Data collected from a large enough sample
    • Data collected at multiple points in time
    • Good quality, reliable data
  • Metrics

    Discussions of Metrics for Data Quality

    • Number of data tracking sheets
    • Completeness of the data
    Some metrics you could use to measure the data collection effort include:
    • Did the volunteer data collectors complete their shift?
    • What challenges and successes did they have in collecting data?
    • Are they willing to continue collecting data for the school?
    • Did the volunteer data collectors represent the diversity of the school?
  • Interviews & FAQs

    Complete the rest of your evaluation form now, or download the training evaluation form (22KB) to access a form.

    Email the form to peph@niehs.nih.gov
  • Special Thanks

    Sally Bond - The Program Evaluation Group, LLC

    Natushia Harris - Duke University Nicholas School of the Environment

    Tiffany K. Harrison - North Carolina Partnership for Children

    Gretchen Kroeger - Duke University Superfund Research Program

    Audrey Pinto - National Institute of Environmental Health Sciences

    Tracey Slaughter - UNC Chapel Hill Superfund Research Program

    Kimberly Thigpen Tart - National Institute of Environmental Health Sciences

    Elizabeth Witherspoon - Comm & Sense

    Dawayne Whittington - Strategic Evaluations, Inc.

    Mary Wolfe - National Institute of Environmental Health Sciences

    Christie Drew (Trainer) - National Institute of Environmental Health Sciences

    Kristi Pettibone (Trainer) - National Institute of Environmental Health Sciences

    Special Photo Credits on Slide 6 to:

    Image 1 and 2 - University of Cincinnati Anti-Idling Campaign Graphic designed by Hamilton County Department of Environmental Services and bus driver training workshop photo credit to Cynthia Eghbalnia, Co-Investigator, Cincinnati Public Schools

    Image 3 - UCLA Goods Movement Community Publication by Impact Project  

  • Special Thanks

    If you have questions or feedback, please contact peph@niehs.nih.gov .
    Christie Drew

    Christina (Christie) Drew, Ph.D. 
    Branch Chief
    Program Analysis Branch
    National Institute of Environmental Health Sciences

    Kristi Pettibone

    Kristianna (Kristi) Pettibone, Ph.D. 
    Health Science Administrator
    Program Analysis Branch
    National Institute of Environmental Health Sciences

Back to Top