Evaluating Public Health Programs

Evaluating Public Health Programs

Sharon Welburn, PhD

Learning Objectives

Describe the steps for program evaluation

Construct a logic model

Design a program evaluation plan

Differences between Research and Evaluation

Adapted from Kogan, M. D. with HRSA

Differences between Research and Evaluation

Adapted from Kogan, M. D. with HRSA

Differences between Research and Evaluation

Adapted from Kogan, M. D. with HRSA

Step 1: Assess Context

What:

• Understand environment, conditions, and systems that shape program
• Examine community needs, priorities, assets, and barriers
• Identify political, social, cultural, economic, and organizational factors

Why?

• Ensures evaluation is relevant and responsive to real-world conditions
• Promotes equity-centered evaluation by recognizing social determinants and power dynamics
• Strengthens interest holder engagement and trust by showing understanding of community context

Step 2: Describe Program

What:

• Describe the purpose of program, how program works, intended outcomes

Why

• Identify the need(s) of the program in the first place
• Understand how program works - pieces and as a whole
• What resources and activities are needed?
• What is the relationship between program elements and expected changes?
• What context(s) and larger system(s) does the program operate?

Step 3: Focus Evaluation Questions and Design

What:

• Determine purpose of evaluation, evaluation questions, methods, agreements, and uses/users of evaluation
• Gain insight? Change practice? Assess effects?
• Plan, plan, and more planning!
• Design, methods, data sources, data collection/ analysis?

Why?

• Quality in = quality out
• Keep evaluation on track
• Ensure feasible, useful, cost-effective, politically viable

Step 4: Gather Credible Evidence

What:

• Collection information (data, evidence) that is relevant, credible, and meaningful
• Follow your evaluation design (see Step 3)
• Are indicators still valid and reliable?
• Are you collecting evidence from multiple sources?
• Quantity =/= Quality
• Consider logistics, timing, human subjects

Why?

• Quality in = quality out
• Enhance utility, accuracy, validity
• Ensure data integrity, confidentiality

Step 5: Generate and Support Conclusions

What:

• Make claims (judgments) based on data that is of merit, value, and significance
• Evidence needs to be aligned with agreed upon values and standards per interest holder input

Why?

• Ensure systematic interpretation, appropriate comparisons
• "Recommend" actions

Step 6: Act on Findings

What:

• Design, preparation, feedback, and follow-up
• Plan for all potential findings - the good and the bad
• Remind of purpose/use, ensure findings are not ignored
• Facilitate insights into action
• Disseminate
• Confider format, timing, style, tone, message source

Why?

• Full disclosure, impartial reporting
• "Recommend" actions
• Ride the wave of other benefits - clarify values, foster teamwork...

Overview of Standards

Adapted from Kogan, M. D. with HRSA

Any questions thus far?

Case Study Example

Implementing and Evaluating a Community-Based Fitness Program

Your agency just completed a community needs assessment with key stakeholders. The results indicate that a rural community would like to implement an intervention to increase fitness levels among adult women. Your agency identified an evidence-based approach that may work in your community: Strong Hearts, Healthy Communities.

What information should they have been gathering for a needs assessment on this topic?

Case Study Example

Implementing and Evaluating a Community-Based Fitness Program

Your agency just completed a community needs assessment with key stakeholders. The results indicate that a rural community would like to implement an intervention to increase fitness levels among adult women. Your agency identified an evidence-based approach that may work in your community: Strong Hearts, Healthy Communities.

Who are some interest holders in the community to plan, implement, and evaluate the fitness program. What issues would you initially address with them?

Step 2: Describe Program

What:

• Describe the purpose of program, how program works, intended outcomes

Why

• Identify the need(s) of the program in the first place
• Understand how program works - pieces and as a whole
• What resources and activities are needed?
• What is the relationship between program elements and expected changes?
• What context(s) and larger system(s) does the program operate?

Step 2: Describe the Program

This step describes the purpose of the program, how the program works, and intended outcomes.

Key Elements

• Goals
• Objectives
• Logic Model
• Types of Evaluations

Info

Info

For an example, click

Better -

• By 2020, decrease the number of children in grade 5 who have one or more cavity as detected by the school surveillance program by 5%.

https://freshspectrum.com/babys-first-logic-model/

In other words...

Logic Models

• How day-to-day activities will lead to desired outcomes
• Theory of change

"The program logic model is defined as a picture of how your organization does its work - the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities / processes and the theoretical assumptions / principles of the program."

Why?

• Helps you plan, implement, and evaluate your program
• Helps you explain to staff, funders, and other interest holders what you are doing and why
• It's your treasure tool and road map

http://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

Logic Model: Main Components

Think "If... then" ... statements

NACCHO W.K. Kellogg Foundation Logic Model Development Guide

Example: Planning a Family Vacation to Chicago!

NACCHO W.K. Kellogg Foundation Logic Model Development Guide

Your Program Does Not Exist in a Vacuum - Other Considerations

Adapted from Kogan, M. D. with HRSA

The Mytown Example

"The folks in Mytown, USA, are striving to meet the needs of growing numbers of uninsured residents who are turning to Memorial Hospital’s Emergency Room for care. Because that care is expensive and not the best way to offer care, the community is working to create a free clinic.”

NACCHO W.K. Kellogg Foundation Logic Model Development Guide

NACCHO W.K. Kellogg Foundation Logic Model Development Guide

Where does formative, process, outcome, and impact evaluation fit?

NACCHO W.K. Kellogg Foundation Logic Model Development Guide

Case Study Example

Implementing and Evaluating a Community-Based Fitness Program

Your agency just completed a community needs assessment with key stakeholders. The results indicate that a rural community would like to implement an intervention to increase fitness levels among adult women. Your agency identified an evidence-based approach that may work in your community: Strong Hearts, Healthy Communities.

Work in pairs - 20 minutes1. Develop an initial logic model for your community program. Include at least 2 goals. Aim to develop 2-3 SMART objectives. 2. Identify data you would collect to determine: 1) if processes occurred, and 2) the effectiveness of your program. What are sources for the data? What general data analysis could you conduct?

Case Study Example

Implementing and Evaluating a Community-Based Fitness Program

Your agency just completed a community needs assessment with key stakeholders. The results indicate that a rural community would like to implement an intervention to increase fitness levels among adult women. Your agency identified an evidence-based approach that may work in your community: Strong Hearts, Healthy Communities.

How could we disseminate the results of our program evaluation?

Now - Let's See How This Applies to Your Project

On Your Own

We'll take 15 minutes for this.

• Based on your Modules 1 & 2 project, write goals and objectives for your program/intervention.
• Sketch a logic model

On Your Own

We'll take 15 minutes for this.

• Based on your Modules 1 & 2 project, write goals and objectives for your program/intervention.
• Sketch a logic model

Discuss with a classmate

We'll take 10 minutes for this.

• Find someone in the class who has a project opposite of yours
• Ex: If you're project is on screening, find someone focused on risk reduction.
• Exchange goals, objectives, and logic model.
• Provide feedback to one another.

Questions?

Objectives

• Steps/activities that you will measure that directly relate to your goal
• Well-defined and clear
• Action-oriented verbs

Goal 1

Objective 1aObjective 1b

Should be SMART: Specific, Measurable, Achievable/Attainable, Relevant, Timely/Time-Bound

Goals

• Overall purpose of program
• Broad, brief statement of end result (think long-term)
• Non-specific, non-measurable (typically no baseline or target date)
• Abstract and difficult to measure
• Specifies direction of effect for target population
• Larger vision in which your objectives will relate

Examples

• Promote quality of life, healthy development, and healthy behaviors across all life stages
• Decrease unintended pregnancy among teens in Hillsborough County, FL

Examples - Healthy People 2020

Goal: Increase the proportion of women delivering a live birth who received preconception care services and practiced key recommended preconception health behaviors

Objective: By 2020, increase the proportion of women delivering a live birth who had a normal weight (BMI 18.5-24.9) prior to pregnancy by 10%.

Objective: By 2020, increase the proportion of women delivering a live birth who took multivitamins / folic acid prior to pregnancy by 10%.

Objective: By 2020, increase the proportion of women delivering a live birth who did not drink alcohol prior to pregnancy by 10%.