|
Evaluating
Your Efforts
Overview
Evaluating your health planning efforts can improve
upon existing programs, assist with planning future
efforts, make the case for funding needs or refocusing
of resources, and add a sense of accountability to your
programs. You can evaluate the conceptualization and
design of a program, monitor program implementation,
or assess the program's effectiveness and efficiency.
Despite the reason for evaluation or the type of evaluation,
the most important thing to remember is that your evaluation
plan will need constant modification depending on the
way the program evolves, the types of data you have
access to, and shifts in stakeholder interest. Also
important to remember is that you are not alone. RC/EZ/ECs
can take advantage of the evaluation expertise of local
health departments or universities or follow one of
many helpful program evaluation guides available from
the United States
Department of Housing and Urban Development (HUD),
the United States Department
of Agriculture (USDA), the United
States Department of Health and Human Services (HHS),
United
Way, and other sources.
(back to top)
Stories
and Models from the Field
The Neighborhood Violence Prevention Collaborative:
Partnering with a University to Evaluate Success
Flint, Michigan EC (Round I)
The Flint, Michigan EC made use of its partnership
with the University of Michigan School of Public Health
to evaluate the success of their Neighborhood Violence
Prevention Collaborative (NVPC). The NVPC was established
to build the capacity of neighborhoods and residents
to address the underlying causes of violence and create
an environment that promotes peace.
To evaluate the NVPC's success, this EC chose to utilize
the evaluation expertise of the neighboring University
of Michigan. Faculty from the University of Michigan's
School of Public Health are in the process of conducting
a local evaluation of these efforts. According to Dr.
Tom Reischl, Associate Research Scientist at the School
of Public Health, surveys conducted both before and
after the EC's neighborhoods receive funding are being
used to indicate the relative growth and stability of
each one's capacity to accomplish their violence prevention
goals. These surveys ask questions regarding the number
of members, the number of participants in meetings,
whether or not any political connections have been developed
and with what organizations the neighborhood has developed
partnerships. Success is measured as growth or stability
in size, in the number of activities conducted or in
the number of available resources.
Another study, currently being conducted by Dr. Katherine
Alaimo, uses case studies to determine if urban gardening
efforts are making a difference in perceptions of safety.
Dr. Reischl is also conducting household surveys that
will be evaluated using mapping techniques. These will
provide a descriptive evaluation of the community's
neighborhoods and help to determine the impact that
the EC is having on non-EC neighborhoods.
Lewiston, ME EC (Round I)
To ensure that they would be able to accurately evaluate
the outcome of their efforts, the Lewiston, Maine EC
conducted a needs assessment that used specific measures
of community access to health. These included current
access to primary care providers and hospital usage
rates. This survey was conducted onsite during Lewiston's
"Night in the Park" event. To obtain a more representative
sample of surveys, the EC chose to redistribute the
survey to members of its Community Outreach Committee.
These citizens are involved in senior citizen organizations,
job training and other community efforts. By building
these measures into their assessments and sampling across
the community, the Community Outreach Committee hopes
to have not only a more accurate assessment of need,
but a better reading against which to measure progress.
Burlington, Vermont EC (Round I)
In the past year, the Community Outreach Partnership
Center (COPC) of the University of Vermont completed
the Neighborhood Association and Quality of Life Canvassing
Research Study, which showed that the work of Public
Safety Project and neighborhood associations have a
positive effect on the quality of life in the Old North
End. Of the residents interviewed, those who live on
a street with a block association were considerably
more likely than those who live on a street without
a neighborhood association to agree with each of the
following statements: "My block is a good place for
me to live", "My block is a comfortable place to live
in without too much noise, traffic or disruptive neighbors",
and "People of all ages are safe living on this street".
A full report on this research study and other studies
undertaken by the COPC are available.
(back to top)
Tools
Evaluation: A Systematic Approach.
Peter H. Rossi and Howard E. Freeman. Sage Publications
(6th edition, 1999). Provides a step-by-step process
for doing evaluations and measuring the impact of social
programs. The book can be purchased from Sage
Publications.
Worksheet I: Potential Health Measures
Worksheet II: Setting Targets for Objective
Worksheet III: Measuring Progress
Worksheet I: Potential
Health Measures
The following is intended to assist you in identifying
different types of measures for your RC/EZ/EC's health
plan. It is not meant to be an exhaustive list, but
provides types of measures many communities have found
beneficial in developing and monitoring health objectives.
|
COMMUNITY
MANAGEMENT
|
HEALTH BEHAVIORS
|
Interagency networks
Open city council meetings
Planning - economic development, social planning
council
Policy environment
Readiness - fire escape plans, CPR training,
retirement preparation
Representation in community groups Responsiveness
- emergencies
Volunteerism level
Voter turnout |
Exercise levels
Overweight prevalence
Tobacco use
Alcohol use/abuse prevalence
Substance abuse treatment need |
|
DEMOGRAPHICS
|
HEALTH CARE RESOURCES
|
Age distribution
Education levels
Median income
Occupations
Population stability
Poverty levels
Unemployment rates |
Insurance status
Medicaid/Medicare providers
Managed care penetration |
|
GROWTH AND NUTRITION
|
HEALTH CARE UTILIZATION
|
Breastfeeding prevalence
Developmentally delayed children
Fruit and vegetable consumption
Disability prevalence
Enrollment in entitlement programs
Elders who participate in fitness programs
Life expectancy
Self-reported health status
Women, Infants and Children (WIC) |
Hospital use rate
Preventable hospitalizations rate |
|
MORTALITY
|
HEALTH OF EMPLOYEES
|
Infant mortality - neonatal,
post-neonatal
Major killers - CHD, cancer, stroke, homicide,
suicide, motor vehicle injuries, unintentional
injuries, diabetes, COPD, AIDS
Overall and age-level |
Sick days used
Workmen's compensation claims
Worksite injuries and deaths |
|
PHYSICAL ENVIRONMENT
|
HEALTH OF MOTHERS AND CHILDREN
|
Environmental conditions -
air, water, recreational water site quality
Environmental hazards
Epidemics
Household smoke detectors
Households on water and sewage treatment systems,
septic systems
Household fuel efficiency
Household recycling
Industrial waste recycling
Lead paint housing vulnerability, soil
Natural disasters
Nuisance index - noise, dirt, odors |
Contraceptive services and need
Low birth weight babies percent
Prematurity prevalence
Prenatal care percent
Teen parenting prevalence |
|
MORBIDITY
|
PREVENTIVE MEASURES
|
Dental caries
among children
Communicable diseases rates
Vaccine preventable disease/deaths
Mental illness prevalence |
Blood pressure
checks
Childhood immunization rates
Cholesterol checks
Colon cancer screening prevalence
Diabetic eye and foot exams
Flu vaccine use among the elderly
Mammography screening prevalence
Pap test prevalence |
|
SOCIAL SUPPORT MEASURES
|
Bike path mileage
Recreation center use
Child abuse investigations
Domestic violence services
Family and friend support networks
Religious center use
Law enforcement
Neighborhood Watch Programs
Self help group participation
Suicide prevention services
Transportation services |
|
Source: Empowerment Zone/Enterprise Community
Health Benchmarking Project. Public Health Foundation,
1998-1999.
Worksheet II: Setting
Targets for Objectives
One of the central issues many communities struggle
with when developing objectives is how to set achievable,
realistic targets for outcome, performance, and process
objectives. The guidance below focuses primarily on
setting targets for health outcomes and performance.
Using an absolute percent decline
Absolute percent decline is based on "best guesses"/expert
opinion. Calculations can be made based on the percent
of the target population reached and change expected.
For example, a decline in mortality of 30 percent expected
in two-thirds of the women with breast cancer.
|
[Start Amount x (1-.30)
x 2/3] + [Start Amount x 1/3] = End
Amount Example: Breast
cancer rate of 33/100,000
[33 x (1 - .30) x 2/3] + [33 x 1/3] = 15.4
+ 11 = 26.4/100,000
|
|
Using peer communities
You can set targets by comparing your community to others
like it. Age and poverty distribution and population
size and diversity may define peer communities. The
following may be used to describe one's peers: typical
values for a specific objective, means or medians, or
the variation among peers. Visit the Public Health Foundation
web site for more information on the Community Health
Status Indicators Project, which is utilizing this strategy:
Click here.
Using the pared-mean method to set data driven benchmarks
The pared-mean method determines "top performance."
This is defined as the best outcome accomplished for
at least 10 percent of the population.
Steps to Compute the Pared-Mean (The article
cited below uses an example of mammography screenings)
- Rank order providers or other units of analysis
(e.g., health departments, jurisdictions) in descending
order of adherence. In this example, metropolitan
statistical areas were ranked according to average
mammography rates.
- Order providers in descending sequence until you
have a subset that equals or exceeds 10 percent
of all patients in the survey. In this example it
was 10 percent of women over the age of 50 in the
survey.
- Calculate the benchmark based on the subset of
units analyzed, dividing the total number of patients
in the subset receiving the recommended intervention
(e.g., mammography screenings by the population).
Source: Allison J., Kiefe C.I., Weissman N.W.
"Can Data-Driven Benchmarks be Used to Set the Goals
of Healthy People 2010?" American Journal of Public
Health, 89(1):61-5, 1999.
What if areas in the community has already achieved
or surpassed their target for an objective?
You can calculate a new, higher target that will be challenging
for local areas that have achieved or surpassed their
original target. You also may wish to note in your health
plan where you have not achieved your previous targets
and redouble your efforts in these areas as well.
Setting targets for process objectives
Many times, communities will put process objectives,
particularly those that pertain to infrastructure (i.e.
data systems, workforce, and research), into place.
These should be examined carefully by the RC/EZ/EC to
determine their applicability to its overall health
goals. Setting measurable targets for process objectives
requires judgment and is not an exact science. To set
process targets, planners should consider the current
status (baseline) of the RC/EZ/EC's public health infrastructure,
seek stakeholder input on the desired level of improvement,
and make a realistic assessment of what can be accomplished
given the community's experience, resources, political
opportunities, and partner commitment.
Using performance measures (http://www.phppo.cdc.gov/nphpsp/)
"Performance measurement responds to the need to ensure
efficient and effective use of resources, particularly
financial resources. It links the use of resources with
health improvements and the accountability of individual
partners." (Prevention Report, Winter 1997) This
is of particular importance since the inception of the
Government Performance and Results Act of 1993, which
aims at holding Federal agencies accountable for spending
public dollars. This extends to states, local jurisdictions,
and other organizations that receive Federal funding.
Performance measures can be incorporated into or based
upon RC/EZ/EC objectives. Please see the following pages,
as well as the CDC's National
Public Health Performance Standards Program for
more detailed descriptions of setting performance measures.
Setting Performance Measures Step
by Step
These examples are based on the State of Maryland's
Healthy Maryland 2000 document
| Step |
Example |
| 1. Relate the performance
measure to an important national, state, or
local health priority area. |
Maryland has undertaken work
related to the national health objective to
reduce coronary heart disease deaths to no
more than 100 per 100,000 people. |
| 2. Measure a result that
can be achieved in 5 years or less. |
Maryland has identified an achievable
result that is linked scientifically to the
Healthy People 2000 Heart Disease and Stroke
priority area: Increase the proportion of
people who engage in light to moderate physical
activity to at least 30 percent of the population. |
| 3. Ensure that the result
is meaningful to a wide audience of partners. |
Target partners are essentially
all Marylanders, with an emphasis on school-age
children and people at high risk for diseases
and medical conditions associated with physical
inactivity (for example, persons with hypertension
and high cholesterol). Partners include principals,
teachers, students, parent-teacher associations,
the state education department, state and
local health and recreational agencies, public
health and medical professionals, and others. |
| 4. Define the strategy
that will be used to reach a result. |
The state of Maryland has selected four
strategies:
- Implement a combination of strategies
that include consumer education and skills
development, health assessment, professional
training, and environmental changes.
- Reinforce risk reduction messages and
promote programs and policies in schools,
work-sites, faith communities, and other
settings.
- Focus on youth and families so that
healthy habits are started early and nurtured
in the family.
- Use a health promotion approach tailored
to reach diverse ethnic and socioeconomic
groups.
|
| 5. Define the accountable
entities. |
The accountable entities depend
upon the strategies selected and the way in
which a particular community is organized.
For Maryland's strategy 2, these entities
include schools, work sites, and community
centers. For example, the Cecil County Public
Schools have agreed to be accountable for
specific tasks related to strategy 2 and are
working in partnership with the Cecil County
Health Department to offer healthy lifestyle
programs to elementary school children. The
programs, such as the Heart Challenge Course,
bring teachers and food service workers together
to promote healthy eating habits and physical
fitness through educational games, classroom
projects, and other activities that appeal
to children. |
| 6. Draft measures that
meet statistical requirements of validity
and reliability and have an existing source
of data. |
In consultation with biostatisticians
and epidemiologists, organizations can draft
measures that are statistically sound. One
of Maryland's performance measures might be
"Increase to 30 percent the proportion of
students in each Cecil County elementary school
who engage in light to moderate physical activity
for 30 minutes or longer every school day
by participating in school physical fitness
activities." |
|
Source: U.S. Department of Health and Human
Services. "Improving the Nation's Health with Performance
Measurement." Prevention Report, 12(1):1-5, 1997.
http://odphp.osophs.dhhs.gov/pubs/prevrpt/97winfoc.HTM.
Worksheet III:
Measuring Progress
Annual Percent Change
This measure can be used to track whether progress is
on course and determine if the RC/EZ/EC's objectives
will be reached. It provides the amount of decline each
year that is needed to reach the target.
Formula:
 |
|
Example Data Showing Percentage Change
Needed to Reach Health Planning Goal
|
| |
Year |
Rate |
| Target |
2010 |
7/1,000 |
| Baseline |
2000 |
10/1,000 |
|
|
Calculations:
 |
| A decline of 3.5% per year between year 2000 and
2010 is needed to reach the target. |
Measuring Progress
This equation is used in measuring progress for each
objective, adapted from Healthy People 2000 Midcourse
Review and 1995 Revisions:
 |
| Note: You will get a negative percentage
when the baseline has gotten worse. |
 | These technical assistance resources for RC/EZ/ECs were
funded by the Office of the Assistant Secretary for Planning and Evaluation,
U.S. Department of Health and Human Services (HHS), through a cooperative
agreement administered by the Health Resources and Services Administration
(HRSA), and prepared by the Public Health Foundation. Duplication
and adaptation, with credit, are encouraged. |  |
|
(back to top)
Tips
for RC/EZ/ECs
- Evaluation is not an end task, but an ongoing task
that should constantly feed the process.
- Plan your evaluation at the outset and obtain baseline
data.
- Establish a formal feedback system.
- Consult experts at your local health department
or university on evaluation mapping.
- Assign responsibility for the evaluation.
- Consider using an outside evaluator.
- Decide who will have access to evaluation information.
- Use both process and formative evaluation.
- Collect qualitative data (in addition to quantitative
date) to tell you why things happened.
- Maintain consistency of terms and data definitions.
- Look into your data needs - what's out there and
what do you need? Use local hospitals, public health
agencies, and colleges and universities to meet your
data needs.
- Partnering businesses can use the Work Opportunity
Tax Credit to hire and train seasonal workers to collect
evaluation information.
- Produce periodic progress reports.
- A good evaluation plan showing results can lead
to future funding.
(back to top)
Links
Evaluation
Tools
Various evaluation tools including a sample anecdotal
record form, protocol for running focus groups and a
sample evaluation report. These tools were created and
sponsored by Georgia Tech.
Evaluation
Activities in Organizations
Used by the Management Assistance Program for Non-profits;
Includes helpful
CDC
Evaluation Working Group: Resources
Centers for Disease Control and Prevention's collection
of excellent links to evaluation resources related to
topics including ethics and standards, step-by-step
manuals, planning and performance tools and reports
and publications.
The
Program Manager's Guide to Evaluation
Written by the Administration on Children, Youth, and
Families, this guide takes program managers through
all the basic steps - from preparation and what to include
in an evaluation through understanding evaluation results.
The Community
Toolbox
The mission of the Community Tool Box is to promote
community health and development by connecting people,
ideas and resources. The web site provides tools needed
to build healthier and stronger communities. The web
site also provides information for those interested
in a variety of community health and development issues
and connects individuals to personalized assistance
for improving community change efforts. The site offers
the following information about evaluation:
Evaluating
Community Programs and Initiatives
Explanations and tools about operations and methods
of evaluation as well as understanding evaluation
results.
(back to top)
|
