HHS Strategic Goals and Objectives - FY 2001 . Appendix E - Program Evaluations

09/29/2000

Program evaluations can play an important role in formulating goals, objectives, and implementation strategies for a variety of planning activities throughout the Department of Health and Human Services (HHS). Program evaluations also tell us whether our efforts are successful. While there are still gaps in what we know, we now are beginning to assemble a body of evaluative information that supports the way we craft our various goals and objectives and substantiates the effectiveness of strategies to achieve those goals and objectives. To illustrate this, we provide a discussion of the evaluative information that contributed to setting our goals and objectives. We also discuss program evaluations that demonstrate the effectiveness of implementation strategies that we will use. Finally, we provide a list of future evaluations that will provide additional insight into the effectiveness of our strategies and cumulative impact of our efforts (1). A goal by goal discussion follows.

 

GOAL 1: REDUCE THE MAJOR THREATS TO THE HEALTH AND PRODUCTIVITY OF ALL AMERICANS

SETTING THE GOAL/OBJECTIVES

A variety of statistical data on health trends in the United States contributed to the creation of Goal 1. For example, information from the National Vital Statistics Report provided the basis for establishing strategic objectives that address major causes of premature mortality and morbidity in the United States. Also useful was a wide variety of information on specific behavioral trends and incidences of disease available from national surveys and public health surveillance systems, such as the Behavioral Risk Factor Surveillance System, the Total Diet Survey (Food and Drug Administration) and the National Household Survey on Drug Abuse (2).

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

Available evaluation studies underline the effectiveness of a number of the strategies that the Department will use to achieve its objectives. For example, a key element in our strategy to reduce tobacco use among youth is the support of tobacco education programs. A recent evaluation of a major anti-tobacco media campaign in Florida demonstrated the effectiveness of education programs in preventing tobacco use, especially when targeted to younger persons. As a deterrent to tobacco sales to minors, the effectiveness of strategies to enforce the prohibition on sales to minors (Synar) is supported by recent evaluations; e.g., a 1998 Battelle study.

Similarly, evaluations, such as the review of the Child and Adolescent Trial for Cardiovascular Health (CATCH) program, point to the effectiveness of education programs in changing behaviors and attitudes toward diet and physical activity (Objective 1.3). Also, findings from Food and Drug Administration's (FDA) Food Label and Nutrition Tracking System indicate that consumers are reading, understanding, and changing their minds about food products as a result of FDA food labeling activities. The positive impact of consumer education on diets is also supported by the study The Effects of Education and Information Source on Consumer Awareness of Diet-Disease Relationships.

Examples of other evaluations that underline the effectiveness of our strategies in Goal 1 include: preliminary results from the National Cross-site Evaluation of High Risk Youth substance abuse prevention programs (Objectives 1.4 and 1.5); results from the National Treatment Improvement Evaluation Studies (NTIES) evaluation showing that treatment works (Objective 1.5); and evaluations of behavior counseling programs such as Project RESPECT (Objective 1.6).

FUTURE EVALUATIONS

Objective Subject Methodology End Date Agency
Objective 1.1
Reduce tobacco use, especially among youth Evaluation of the effectiveness of population-based tobacco prevention and control programs National/state prevalence surveys and demand models based on tobacco pricing and state policies Ongoing with annual updates Centers for Disease Control and Prevention (CDC)
Objective 1.2
Reduce the incidence and impact of injuries and violence in American society Evaluation of state and local school-based programs designed to prevent chronic disease, including dietary patterns, physical activity, and overweight/obesity Quasi experimental school-based studies Ongoing CDC
Evaluation of multi-faceted interventions for community-dwelling elderly To be developed Being developed CDC
  Evaluation of multi-faceted youth violence prevention interventions To be developed Being developed CDC
  Evaluation of interventions to increase proper restraint use for children To be developed Being developed CDC
  Evaluation of the effect of state and local residential smoke alarm legislation on smoke alarm use and reductions in injuries and fire related deaths To be developed Being developed CDC
Objective 1.3
Improve the diet and level of physical activity of Americans Evaluation of consumer knowledge of relationship between diet and health and dietary supplement labels National Sample (telephone survey) 2002 FDA
Objective 1.4
Reduce alcohol abuse and prevent underage drinking Evaluation of the effectiveness of the Hispanic/Latino media campaign to encourage dialogue between parents and children about substance abuse Survey 2001 Substance Abuse and Mental Health Services Administration (SAMHSA)
Objective 1.6
Reduce unsafe sexual behaviors Evaluation of state and local school-based programs designed to improve adolescent reproductive health, including prevention of HIV, other sexually transmitted diseases (STDs), and teenage pregnancy Quasi- experimental school-based studies Ongoing CDC
Objective 1.7
Reduce the incidence and impact of infectious diseases Analysis of influenza and pneumoccocal reports/data Claims Database analysis 2002 Health Care Financing Administration (HCFA)

 

GOAL 2: IMPROVE THE ECONOMIC AND SOCIAL WELL-BEING OF INDIVIDUALS, FAMILIES, AND COMMUNITIES IN THE UNITED STATES

SETTING THE GOAL/OBJECTIVES

Data from a variety of national, state and program-specific sources provided valuable insights and information useful for the development of Goal 2, including development of the objectives and implementation strategies. For example, the national evaluation of welfare-to-work activities provided information on the effectiveness of the JOBS program in seven sites and was the basis for many of the reforms in the Personal Responsibility and Work Opportunity Reconciliation Act of 1996. Additionally, state and program administrative data were particularly useful in assessing trends and establishing the objectives for child welfare, abuse and neglect, early learning (Head Start) and child care. The Department's third annual report, America's Children: Key National Indicators of Well-Being, provided a secondary source of trend data for these objectives. Projections by the Department's micro simulation model, the Transfer Income Model (TRIM), were useful in testing alternative approaches and strategies for human services programs. Census data and data from surveys of the National Center for Health Statistics (CDC) contributed to the development of objectives that address trends in aging and long term care.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

In Goal 2, there are a number of implementation strategies that focus on identifying effective program practices and disseminating these to states and other service providers through Federal technical assistance and capacity development activities. Evaluative assessments of these efforts point to their value and argue for continuing to help identify and disseminate best practices as a key strategy to achieve our objectives in Goal 2. For example, there is substantial evidence that many welfare-to-work experiments supported by the Department have been adopted by states. Many elements of successful demonstrations were adopted in the welfare reform provisions of Temporary Assistance to Needy Families (TANF).

In other areas, we point to the success of bonus payments and technical assistance to help states reduce barriers to adoption (Objective 2.4). Recent evaluative data indicate substantial increases in adoptions in FY 1997 and FY 1998, indicating the success of these strategies. Also, the Government Accounting Office (GAO) recently found that the use of adoption incentives increases the likelihood of adoption of older and minority children and is cost effective (GAO/HEHS-97-73).

There is evaluative evidence going back a number of years that demonstrates the success of working through programs such as Head Start to link children to health care. These include A Descriptive Study of Head Start Health Services (a 1996 study of a representative sample of 1,200 children in 40 Head Start programs), current Head Start Program Information Report data, and Child Care State plans. Similarly, evaluations are beginning to show that providing access to quality childcare is effective in promoting healthy childhood development. (See the 1999 study, Access to Child Care for Low-Income working Families, and a National Institute of Child Health and Human Development study examining outcomes for children attending centers that meet professional standards.) Finally, results from the Family and Child Experiences Survey are beginning to show the positive impact of Head Start on child performance in cognitive and social skills, indicating learning readiness for kindergarten (Objective 2.3).

FUTURE EVALUATIONS

Objective Subject Methodology End Date Agency
Objective 2.1
Improve the economic independence of low income families including those receiving welfare Evaluation of employment retention and advancement strategies; impact of welfare reform on child outcome measures; rural welfare to work strategies; and the effectiveness of serving special populations Experimental and non-experimental Variable 2001-2005 Administration on Children and Families (ACF)
Objective 2.3
Improve the healthy development and learning readiness of preschool children Continuing evaluation of the impact of Head Start (FACES) on the social and cognitive progress of children Surveys, observations, childhood assessments Ongoing ACF
  Continuation ofA National Study of Child Care for Low-Income Families Analysis of administrative data, surveys, interviews 2002 ACF
Objective 2.4
Improve the safety and security of children and youth Continuation of a national longitudinal study of child welfare that looks at the effectiveness of services provided for families and children Surveys, interviews Ongoing in three to five year cycles ACF
  Assessment of child welfare outcomes in areas of safety, permanency, and child and family well-being Surveys, interviews Ongoing ACF
  Evaluation of the impact of family preservation and support services Surveys, interviews Date-phased ACF
Objective 2.5
Increase the proportion of older Americans who stay active and healthy Evaluation of multi-faceted fall-prevention programs for community-dwelling elderly To be decided Being developed CDC
Objective 2.6
Increase independence and quality of life of persons with long-term-care needs Evaluation of the home and community-based services waiver program

Evaluation of multi-state demonstrations for integrating acute and long-term-care services

Descriptive analysis and consumer survey

Quasi-experimental using surveys, case studies and database analysis

2002

2002

Health Care Financing Administration (HCFA)

HCFA

  Multi-state evaluation of dual eligibles demonstrations (cash and counseling demonstrations for making individuals more involved in planning and directing their community-based long-term-care services) Control group 2003 HCFA
Objective 2.7
Improve the economic and social development of distressed communities Evaluation of impact of Individual Development Accounts To be determined 2005 ACF

 

GOAL 3: IMPROVE ACCESS TO HEALTH SERVICES AND ENSURE THE INTEGRITY OF THE NATION'S HEALTH ENTITLEMENT AND SAFETY NET PROGRAMS

SETTING THE GOAL/OBJECTIVES

Data from a number of health related surveys, such as the U.S. Census and Current Population Survey, were instrumental in helping set Goal 3 objectives that address problems such as the lack of access to health care insurance and services and health disparities. Examples include the Medical Expenditure Panel Survey (Agency for Healthcare Research and Quality), the National Survey of Health Insurance (Kaiser/Commonwealth), the National Vital Statistics System (CDC), policy briefs of the National Center for Cultural Competence, and the report of the Surgeon General on Mental Health. Also useful were program data from the Health Resources and Services Administration on medical shortage areas and cost data from the Agency for Healthcare Research and Quality (AHRQ) on the cost of services to persons with HIV/AIDS, (see AHRQ Publication No. 99-RO28). Information from the Department's Office of the Inspector General (OIG) contributed to the development of our objective on the integrity of the Medicare and Medicaid programs. Data from the Medicare Current Beneficiary Survey were useful for assessing issues related to the effectiveness of and access to Medicare services.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

A number of evaluative studies and other evidence illustrate the effectiveness of Goal 3 implementation strategies in increasing access to and effectiveness of health care services. For example, increasing the supply of physicians in under-served areas is a successful strategy for improving access to health care services. In addition, there is a similar impact where community health centers are located, and considerable evidence supports the success of Ryan White programs in increasing access to health care services for persons with HIV/AIDS. Finally, a continuing national evaluation of the strategy to support comprehensive community mental health services for children and their families shows improvements in a range of child outcome indicators (e.g., school attendance and behavior).

FUTURE EVALUATIONS

Objective Subject Methodology End Date Agency
Objective 3.1
Increase the percentage of the nation's children and adults who have health insurance coverage Assessment of the impact of welfare reform on Medicaid

Evaluation of the State Children s Health Insurance Program

Analysis of claim and eligibility files
National analysis of enrollment and service use files and meta-analysis of state evaluations
2001

2004

HCFA

HCFA

  Cost-benefit analysis of the Health Insurance Portability and Accountability Act (HIPAA) and related provisions Modified cost-benefit analysis 2001 HCFA
  Evaluation of the use of High Risk Pools Case study 2001 HCFA
  Evaluation of the treatment of associations under HIPAA reform to determine their effectiveness in extending health insurance coverage Case study 2001 HCFA
  Evaluation of the Qualified Medicare Beneficiary (QMB) and Special Low Income Medicare Beneficiary (SLMB) programs National sample survey, focus groups, case studies 2002 HCFA
Objective 3.2
Eliminate disparities in health access and outcomes Evaluation of Racial and Ethnic Approaches to Community Health (REACH) demonstrations to eliminate health disparities Comparison of standardized scores with and across communities and Behavioral Risk Factors Surveillance System (BRFSS)-matched demographic comparison Ongoing CDC
Objective 3.3
Increase the availability of primary health care services for under-served populations Evaluation of Health Center Performance User visit survey 2001 HRSA
  Evaluation of Critical Access Hospitals program Rural Research Center case studies and analyses 2002 HRSA
Objective 3.4
Protect and improve the health and satisfaction of beneficiaries in Medicare and Medicaid Evaluation of the Medicare+Choice Program, including beneficiary access to managed care options, the cost and quality of services and beneficiary satisfaction Analysis of beneficiary encounter data 2001 HCFA
Objective 3.5
Enhance the fiscal integrity of HCFA programs and purchase best value health care for beneficiaries Evaluation of coordinated care and disease management demonstrations to determine their effectiveness in promoting value-based purchasing for Medicare Claims and cross-site analysis 2005 HCFA
Objective 3.6
Improve the health status of American Indians and Alaska Natives (AI/AN) Evaluation of obesity at diabetes prevention pilot sites to determine effectiveness of prevention approaches in decreasing overweight and obesity in young children Clinical assessments and behavioral surveys 2004 Indian Health Service (IHS)/ACF
Objective 3.7
Increase the availability and effectiveness of services for the treatment and management of HIV/AIDS Evaluation of Ryan White HIV/AIDS programs Analysis of grantee data 2001 HRSA
Objective 3.8
Increase the availability and effectiveness of mental health care services Evaluation of whether target audiences are adopting Treatment Improvement Protocols Mail survey 2001 SAMHSA
Objective 3.9
Increase the availability and effectiveness of health services for children with special health care needs Study on Children with Special Health Care Needs State and Local Area Integrated Telephone Survey (SLAITS) interview mechanism 2002 HRSA, CDC

 

GOAL 4: IMPROVE THE QUALITY OF HEALTH CARE AND HUMAN SERVICES

SETTING THE GOAL/OBJECTIVES

Goal 4 development was substantially influenced by recent findings of the Advisory Commission on Consumer Protection and Quality in the Health Care Industry and the Institute of Medicine's report on medical errors. The Commission found that medical errors occur in hospitals, nursing homes, pharmacies, urgent care centers, and home care, and that all medical errors cost the nation approximately $37.6 billion annually. Also, the challenge to improve health care quality in the United States is well outlined in the Department's report, The Challenge and Potential for Assuring Quality Health Care for the 21st Century. In developing Goal 4, we also considered the continuing need to improve the quality of human services based on widely available trend data on the well-being of children and families in the United States.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

Although much of the initiative to improve care quality is new and evaluations of programs and activities are just beginning, some evidence of effectiveness has emerged. This is linked to the design of our strategies. For example, a key component of our quality improvement strategies is to develop evidence-based findings on effective health services and promote use of the findings. Evaluation findings of the Agency for Healthcare Research and Quality in 1999 (Publication No. 99-R043) and evidence from other studies (Publication No. 95-N012) support the conclusion that evidence-based research is making its way into practice and, in turn, is contributing to improvements in patient outcomes. Similarly, GAO testified in 1995 (GAO/T-HEHS-95-221) that AHRQ practice guidelines seemed to have a positive impact on patient outcomes.

Efforts to increase consumer and patient use of health care information are the focus of an ongoing evaluation of the Health Care Financing Administration education program-Medicare & You-which is designed to help beneficiaries make the best use of new benefits and program flexibility. This evaluation will continue to provide feedback on the program and guide future directions. Finally, ongoing assessments of the impact and effectiveness of the Mammography Quality Standards Act has shown the value of certification and inspection strategies as an effective means of addressing patient protections.

A key implementation strategy for improving the quality of human services programs is the development of a broad framework that includes quality data, performance measurement systems, and program evaluations. As policy and program design devolve to state and local levels, it is vital that these levels of government have reliable information on which to base their decisions and that the effects of different policy and program choices on quality and accessibility is understood. Documenting, understanding, interpreting, and facilitating the exchange of information and experiences among states is essential to providing high quality services that promote the well-being of families and children.

FUTURE EVALUATIONS

Objective Subject Methodology End Date Agency
Objective 4.1
Enhance the appropriate use of effective health services Evaluation of Centers for Education and Research on Therapeutics to assess their effectiveness in translating and disseminating objective information on the appropriate and safe use of therapeutics Citation analysis; other methodologies under development evaluation to commence in FY 2001 FY 2002 (first quarter) AHRQ
  Assessment of State Rules and Practice Regarding Collection and Reporting of Racial and Ethnic Data by Health Insurers and Managed Care Plans Database analyses and selected site visits 9/2001 Office of Public Health Service (OPHS)
Objective 4.2
Increase consumer and patient use of health care quality information Expanded evaluation of Medicare & You Handbook: 2000 (the national education campaign to help Medicare beneficiaries make choices among health benefits and plans) Survey Ongoing HCFA
  Evaluation of the use of web sites as an effective means of disseminating health care quality information Analysis of consumer feedback data 2001 HCFA
Objective 4.3
Improve consumer and patient protection Evaluation of the use of a quality indicator format in the end stage renal disease survey process
National Assessment of Culturally and Linguistically Appropriate Services in Managed Care Organizations (MCOs) Serving Racially and Ethnically Diverse Communities
Expert assessment

Survey of random sample of MCOs

2003

9/2001

HCFA

OPHS

Objective 4.4
Develop knowledge that improves the quality and effectiveness of human services practice See Goal 2 evaluations See Goal 2 See Goal 2 See Goal 2

 

GOAL 5: IMPROVE THE NATION'S PUBLIC HEALTH SYSTEMS

SETTING THE GOAL/OBJECTIVES

A variety of assessments of the capacity of the Public Health Service to identify and respond to health problems in the United States are available and support the need to strengthen the public health infrastructure. For example, a 1999 GAO study (GAO/HEHS-99-26) documented problems with laboratory capacity. This is supported by a Department assessment (1997), Public Health Workforce: An Agenda for the 21st Century, which singles out laboratory capacity as a pressing problem. Healthy People 2010 documents the need for better information technology. The Institute of Medicine (1988) published perhaps the most comprehensive view of the challenge, The Future of Public Health.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

Achievement of Goal 5 rests largely on the dual strategies of improving the surveillance and response capacity of federal, state, and local health agencies and improving the effectiveness and timeliness of communications throughout the public health system. Our adoption of these strategies is supported by assessments that are beginning to show successes in several areas. For example, assessments of efforts in the National Center for Health Statistics (NCHS) to improve the timely release of surveillance and survey data have been positive. Also, an assessment of the CDC Assessment Initiative to enhance the ability of state and local health departments to use data for policy making has been positive.

In the area of medical device safety, evidence suggests that FDA information dissemination about faulty medical products, transmitted through advisories, has a positive impact on product safety, although further review is indicated. There is evidence that the implementation of FDA Adverse Event Reporting System for Biologics has resulted in improved products through changes in product labeling. Betaseron is one example. The success of food safety consumer education strategies is supported by analytic findings in Background Research and Recommendations for the Food Safety Campaign. Finally, strategies to make drug prescription information more easily available and understandable seem to be successful, according to information obtained through our biennial National Survey of Prescription Medicine Information Received by Consumers.

FUTURE EVALUATIONS

Objective Subject Methodology End Date Agency
Objective 5.1
Improve the capacity of the public health system to identify and respond to threats to the health of the nation's population Evaluation of how the public health infrastructure affects the performance of public health programs and interventions
Evaluation of the impact of new information systems, technology, and informatics training on public health practice

Evaluation of the impact of public health workforce development (training) on public health organization performance and the public health

Evaluation of the effectiveness of safety and public health advisory issuances

Survey

Survey

Survey

Survey

FY 2001

FY 2001

FY 2002

Ongoing

CDC

CDC

CDC

FDA

Objective 5.2
Improve the safety of food, drugs, medical devices, and biological products National Survey of Prescription Medicine Information Received by Consumers

Evaluation of consumer knowledge of food safety issues

Survey

Survey

Ongoing

Biennial
2002

FDA

FDA

 

GOAL 6: STRENGTHEN THE NATION'S HEALTH SCIENCES RESEARCH ENTERPRISE AND ENHANCE ITS PRODUCTIVITY

SETTING THE GOALS/OBJECTIVES

Almost every day, the American health science research community announces new discoveries that hold tremendous potential for the prevention and treatment of disease and injury. The promise of these discoveries argues for the nurture of the research infrastructure that produces the discoveries. As a result of this productivity, strengthening this country's health sciences enterprise has become and remains one of the strategic goals of the Department.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

Success in achieving Goal 6 will rely on how effectively our strategies nourish health research. One element is to facilitate the conduct of research and to move successful research into practice and products. Evaluative information supporting our direction is continuing to emerge. For example, attempts to accelerate the development of new medical products through streamlining the product application and review process has led to shorter review times and we are seeing new products approved under Fast Track processes. (Two such products for the treatment of HIV were approved in 1999.) Overall, streamlining efforts in response to the Prescription Drug User Fee Act (PDUFA) and the Food and Drug Administration Modernization Act (FDAMA) efforts are working to decrease product approval times, as reported in the FY 1999 Performance Report to Congress.

FUTURE EVALUATIONS

Objective Subject Methodology End Date Agency
Objective 6.1
Advance the scientific understanding of normal and abnormal biological functions and behaviors Review and assessment of results achieved from funded research, conducted as a normal part of scientific planning and priority setting Various mechanisms, involving numerous internal and external groups Continuous NIH
Objective 6.2
Improve our understanding of how to prevent, diagnose, and treat disease and disability Review and assessment of results achieved from funded research, conducted as a normal part of scientific planning and priority setting Various mechanisms, involving numerous internal and external groups Continuous NIH
Objective 6.4
Accelerate private sector development of new drugs, biologic therapies, and medical technology Evaluation of statutory performance under PDUFA and FDAMA Administrative Data Analysis Ongoing FDA
Objective 6.5
Strengthen and diversify the base of well-qualified health researchers Evaluation of Minority Institution Research Development Programs awarded as cooperative agreements
NRSA Postdoctoral Evaluation Study

Survey of graduate science student support
Survey of doctorate recipients

Modified case study approach

Longitudinal survey with comparison groups

Survey

Survey

2001

2001

Biennial (2002)

Biennial (2002)

NIH

NIH

NIH

NIH

Objective 6.6
Improve the communication and application of health research results Evaluation of Internet-based tools to improve cancer clinical trials Comparison groups 2002 NIH

Notes:
1. Program evaluation information is displayed only for those objectives for which future evaluations are planned.
2. This includes information tracking major health risks in America (e.g., the percentage of adults who are obese), behavioral risk factors among adults for cardiovascular disease (Centers for Disease Control and Prevention), and information tracking other trends.