Measuring Effectiveness

Measuring Effectiveness

Types and methods of measuring program effectiveness.

Evaluation can be done at any point during the process

The common misconception is that you can only check effectiveness at the end of a process. Different types of evaluations can be done to check the effectiveness of your program at different stages.

EvaluATION Measures

Formative Evaluation

Process Evaluation

Formative evaluation is done at the beginning of a process, or intermittently to ensure that it's on target.

Process evaluation is done during implementation to check effectiveness.

Summative Evaluation

Outcome Evaluation

Summative evaluation is done at the end to reflect on the program as a whole.

Outcome evaluation is done at the end to see if the goals were met.

Economic Evaluation

Impact Evaluation

Economic evaluation can be done at any point to compare costs to revenues, either past or future.

Impact evaluation is done at the end to reveal changes the process inspired.

Evaluation Method:

Evaluation Method:

Formative

Summative

Evaluates the experience intermittently throughout the duration of the program

Evaluates the experience holistically at the end of the program

Are we meeting our goals? Present tense.

Did we meet our goals? Past tense

v.s

Is this timetable reasonable? Are we on track with our timetable?

Did we have enough time to accomplish our goals?

What challenges do we expect?What challenges are we having?

What challenges did we have throughout the program? How can we do better in the future?

Are our goals reasonable? Are our goals reasonable for our progress?

Were we reasonable in our initial project goals?

Logistics

How are we implementing our program? Is our program being implemented properly?

Satisfaction

How satisfied is the program audience?How satisfied are the people implementing the program?

Process

Evaluation

Audience

There are many ways to evaluate a process, but you must keep a few key themes in mind: logictics, overall ease and satisfaction, innovations, and audience.

Is this program reaching the intended audience?What needs is this program fulfilling?

Innovation

What innovations are we able to implement?What innovations are we having to create?

Measuring for

Measuring for

Impact

Outcome

Impacts measure holistic changes to the program

Outcomes measure adherence to expected goals and objectives

Changes include outcomes

Expected Goals v. Actual Outcomes

v.s

Did our process improve the experience or create a benefit?

Does this process solve the initial question/problem?

Did we achieve our goals, or satisfy our purpose?

How close were we to meeting each of our goals?

What anticipated or unanticipated changes occured?

Did we meet enough goals or benchmarks to call this a success?

Economic Analysis

Complex Cost Benefit Analysis

Simple Economic Analysis

A simple economic analysis will compare costs and revenues after the fact on a smaller, more immediate scope. Costs may be compared across time or across items such as facilities or projects.

A complex cost benefit analysis is done when making future decisions and considers obvious costs and revenues, and involves market appreciation and depreciation, rate of return, return on investment, intangibles, and opportunity costs.

CLICK HERE

CLICK HERE

Complex Cost Benefit Analysis

Costs

• Initial Cost: The initial cost of the project that you will incur before the project can begin. These occur in year 0 of the project life
• Staffing Costs: The yearly costs of staffing your project/facility. Staffing costs usually aren't incurred until year 1 of the project life
• Operational Costs: The yearly costs to keep your project going that are unrelated to staffing.
• Opportunity Cost: This is the lost revenues that you would receive if something else were to go in here, calculated by taking the return on investment from the option not taken and subtracting the return on investment from the option taken.

A complex cost benefit analysis will account for intangible factors, and appreciation/depreciation values for future payments/expenses. The next slide will use the new multigenerational recreation center on the plot southwest of Erwin Park as an example to understand the concepts

Revenues

• Yearly Revenue: The annual revenue made by the project.
• Land Appreciation: In Texas, the value of land appreciates about 10% each year. The life of a facility is about 25 years. The future value of land today that will appreciate annually is calulated as compounding percentage of the new value each year.
• Return on Investment: (Final Value - Initial Cost) / Initial Cost
• Final Value takes the Net Present Value (NPV) of each reoccuring benefit per each reoccuring period and subtracts the Future Value of Annuity Due of each reoccuring payment per each reoccuring year.

Complex Cost Benefit Analysis Math

• Yearly Revenue: $447,000
• Land Appreciation: $18,000,000
• Future Value = Initial Value x (Appreciation Rate +1)^Time
• Initial Value: 1,700,000
• Land Appreciation Rate = 10%
• Time = 25 years

• Initial Costs: $60,000,000
• Staffing Costs: $768,000/year
• Operational Costs: $124,000/year
• Opportunity Costs: $318,000/year
• (ROI Potential - ROI Actual)

• APEX Revenue - Expenditures = -127,000
• OSRC Revenue - Expenditures = -445,000
• -127,000 - -445,000 = 318,000

Return on Investment: (Final Value - Initial Cost) / Initial Cost Final Value = (NPV Revenue - NPV Costs) per year ROI = -2.39

Limitations

Different evaluation methods have unique limitations

Formative Evaluation

Process Evaluation

Summative Evaluation

May yield unpleasant results, and make users reluctant to evaluate formatively.

If done lazily, outcomes can look on track even if the project is off course.

Summation often focuses more on the negative aspects than the positive ones.

Outcome Evaluation

Economic Evaluation

Impact Evaluation

Goals must be SMART or the progress is unmeasurable.

The longevity of government projects can make CBAs inefficient, ineffective, or unethical.

Collection methods are suseptable to recency bias and sampling bias.

Understanding the limitations of evaluation methods doesn't make them disappear. Most limitations can be counteracted if they're made apparent early in the process.

How long will this take? What will this upgrade accomplish? What challenges do we expect?

Formative

Are households being transferred properly? Do we know how to run reports? Are we on schedule?

Process

SAMPLE PROCESS

How can we implement a smoother transition next time? What additional thoughts do we have?

Summative

Here are examples of different evaluation methods for our recent upgrade of RecTrac 3.0 to RecTrac 3.1. Click on the + signs to read about what each evaluation might look like for the RecTrac update.

Based on our goals, was this upgrade successful? Did we meet our timetable?

Outcome

How much will this upgrade cost? Will it save money? Will it cost staff efficiency time?

Economic

Impact

Is staff satisfied with this upgrade? Did we benefit? How? Does this change positively impact staff?

Cost Benefit Analysis

Complex

I want to open a new recreation center on the parkland southwest of Erwin Park.

• Construction Cost: 60 million, one time
• Yearly Membership Revenue: 447,000/year
• Yearly Staffing: 768,000/year
• Operation Budget: 124,000/year
• Opportunity Costs: 318,000/year
• Land Appreciation: FV = 18million @ 25y
• Return on Investment: (Final Value - Initial Cost) / Initial Cost = -2.39

Economic Analysis

Simple

I want to see how our different centers are doing economically.

• OSRC Differential: $-445,000
• OSRC Expenditures: $892,000
• OSRC Revenue: $447,000

• APEX Differential: $-127,000
• APEX Expenditures: $5,677,000
• APEX Revenue: $5,550,000

• Conclusion: OSRC recoups about 50% of their expenditures whereas APEX recoups about 98% of their expenditures.