Performance Improvement 1997. Future Directions in HHS Evaluation


Last year's annual report, Performance Improvement 1996, stated that HHS agencies, ASPE, and OPHS would focus future evaluation efforts in three directions: (1) the impact of transformations in health and human services; (2) the development of performance measures; and (3) overall program performance improvement. In FY 1997, those directions continue to be the principal evaluation priorities for HHS, and its evaluation resources will be used to generate new knowledge to improve the performance of its programs.

Impact of Transformations

The Secretary remains committed to an evaluation strategy to examine the transformations now taking place in health and human services and the impact of those changes on the well-being of Americans--especially the vulnerable populations that are a high priority for HHS programs, such as disadvantaged or low-income children and families, the elderly, racial and ethnic minorities, and individuals with disabilities.

The financing and delivery of health care in the United States is being transformed by the growth and evolution of managed care and by other innovations in the financing, management, and delivery of health services. HHS agencies need to better understand how these transformations directly affect their programs. For example, HCFA is now developing a comprehensive monitoring and evaluation plan to examine the Medicare and Medicaid programs on such dimensions as access to care, quality, efficiency, cost, and beneficiary satisfaction. They will implement new demonstrations that test alternative payment and health care delivery models and evaluate the results.

Public health programs supported by HHS are an essential part of the health care safety net for the uninsured and a vital element in protecting the health of all Americans. For HHS, an increasing challenge is to understand how managed care and other financing and service delivery innovations affect public health systems, public health programs, and the clients those programs serve. For example, HRSA and SAMHSA have already made significant investments to study the effects of new financing and service delivery mechanisms, especially managed care, on their programs, services, and clients. In upcoming years, ASPE will complement their efforts by undertaking additional evaluations focused on the public health system and programs. Particular emphasis will be given to evaluation issues that address the public health infrastructure; how changes affect vulnerable populations served by the public health system; and how program transformations taking place outside the health arena, such as welfare reform, interact with and affect the public health care system. In addition, HHS will continue to support evaluations to determine the effect of managed care on individuals with disabilities who have both health and long-term care needs.

In parallel, human services programs are undergoing transformations in their purpose, organization, and financing. Enactment of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 eliminated individual entitlement to cash assistance and replaced it with a fixed block grant to States. The law also placed a 5-year time limit on benefits, imposed strict work requirements on recipients, reduced benefits and services available to legal immigrants, and greatly expanded States' authority over welfare programs. The Nation has a great deal at stake in the success of this program, and timely and careful evaluation will be important to help ensure that success.

Studies are needed to evaluate the combined effects of changes taking place in health care delivery systems and in income supports for low-income children and families who often have special needs. For example, one issue is how to help substance abusers or victims of domestic violence whose entitlements and access to services will change under the terms of new welfare reform legislation. Under the new law, welfare recipients generally will be required to work to receive benefits and will face overall time limits on welfare benefits. In the past, many clients identified as substance abusers or victims of domestic violence were not required to participate in work activities. While some of these clients will be exempt under the new law, others must enter the workforce. Case managers need better tools to assess clients' barriers to sustaining employment and to assist these clients in addressing those barriers.

Children's issues will also be a priority. HHS will identify and evaluate effective tools--including methods of linking children and families under the new welfare system with health and family support services--and on understanding the extent and manifestations of health problems among the welfare population.

The interaction of health and welfare changes also raises questions about the impact of welfare changes on the health status and well-being of selected populations of disabled children, low-income mothers with disabilities, and elderly people with health and long-term care needs. For many in these groups, access to health care and supportive services was previously dependent on entitlements that may no longer exist. Studies are needed to examine how these populations obtain necessary health and long-term care services once their entitlements are terminated, as well as the extent to which their health status and quality of life changes. For example, the AoA is interested in examining how State aging agencies use funds under the Older Americans Act, along with other sources of funding, to develop comprehensive and coordinated systems of community-based services for the elderly. These issues become particularly important in light of the demographic changes facing the Nation as the population ages.

Development of Performance Measurements

Implementation of GPRA requires HHS to focus a substantial portion of its evaluation activities on the development of performance measures for its programs and on information systems necessary to produce the data expected to support GPRA's performance measurement requirements. By the end of FY 1997, HHS will have in place a departmentwide strategic plan. The plan will contain the HHS mission statement, the long-term strategic goals, and descriptions of how HHS intends to achieve those goals with its program resources. Next, HHS will develop its annual performance plan for FY1999, which will link the HHS strategic goals to specific program activities. Beginning with FY1999, the third major GPRA element, the Program Performance Report, will present performance data related to the strategic goals based on the previously identified performance indicators.

The performance plan for FY 1999 will identify the performance measures that HHS agencies and offices will use to assess program performance related to achieving the goals. GPRA has several expectations for these measures:

  1. They should to be tied to program goals and demonstrate the degree to which the desired results are being achieved.
  2. The number of measures for each goal should be limited to avoid confusion in assessing accomplishments and making adjustments.
  3. Measures should be responsive to multiple priorities, such as quality, cost, customer satisfaction, and stakeholder concerns.
  4. Performance measures should be linked directly to the offices responsible for making programs work in order to reinforce accountability.

Since GPRA was passed in 1993, HHS agencies have utilized evaluation projects to develop performance measures related to its health and human services programs. For example, AHCPR has initiated projects to develop and test the validity and reliability of agencywide performance measures of its health services research grants process and dissemination mechanisms. CDC now has projects to develop indicators to assess performance of its HIV-prevention community-planning programs and its State-based diabetes control cooperative agreements program. HRSA has invested its evaluation resources in training and technical assistance to program managers to develop measures for GPRA implementation. Featured in chapter II is an HRSA report about the efforts of the Bureau of Health Professions to develop a set of outcome-based performance measures and a plan for a monitoring system to support program management.

In addition to performance measures for HHS programs, the Federal grant relationships to States in public health are evolving into health outcomes-based performance management. For the past two years, ASPE and OPHS have collaborated on several cross-cutting efforts to involve States, communities, and service recipients in identifying such a data system. A major project with the National Research Council is identifying measures that can be used jointly by HHS agencies and States to measure their shared performance in grant management. The project will also recommend specific steps that can be taken to improve State and local community data capabilities. A related project is developing approaches to obtain comprehensive baseline and trend data on public health infrastructure. A third study is building on a Robert Wood Johnson Foundation project to assess the quality of data that States collect and determine whether there is a foundation on which to build a multi-State health data system that can be shared among States to improve health policy decision making.

HHS will continue to invest its evaluation resources in performance indicators to ensure a successful implementation of GPRA. The evaluation strategies of HHS agencies, mentioned in chapter III, include the priorities of projects that examine program performance objectives and develop useful measures of program outputs and outcomes. GPRA offers HHS agencies an opportunity to develop performance measurement systems that will eventually link program evaluation activities to budgeting. HHS's evaluation set-aside authority, such as the 1-percent authority for some PHS agencies, is an important resource to help program managers identify performance objectives and test the validity and reliability of progress indicators.

Performance Improvement

Program managers must initiate evaluations useful for improving the performance of HHS programs and ensuring that program operations are efficient and effective. Evaluations are an essential resource for HHS's Continuous Improvement Program, aimed at supporting development and operation of information systems and special studies to enable program managers to measure customer satisfaction with HHS services.

Several in-progress or planned projects illustrate HHS's evaluation priority of continuous improvement of services. AHCPR is initiating a study to examine the Medical Expenditure Panel Survey regarding the efficiency of its new design and its effectiveness in answering key health policy questions. They will look at the survey process, including data support contracts and the capacity of the survey data for developing health services analytical models.

FDA is examining its program to assist small businesses in complying with FDA rules, which includes determining FDA's success in coordinating different sources of agency advice and the appropriateness and timeliness of the advice. The evaluation will consider the training and materials available to FDA staff to help them answer questions from small businesses, identify barriers to timely and useful responses, and develop procedures for tracking inquiries.

One of HRSA's priorities is to improve access to the continuum of care for persons with HIV/AIDS through the Ryan White Comprehensive AIDS Recovery Emergency (CARE) Act programs. One project concerns the management of persons infected with both HIV and tuberculosis (TB) in metropolitan areas to identify models of successful collaboration between HIV/AIDS and TB control programs. The purpose is to determine the extent to which Ryan White CARE Act grantees are offering guidance to providers on TB screening and prophylactic and therapeutic regimes.

NIH is currently conducting a survey of individuals who applied to NIH for a research grant in FY 1992. The study will obtain information on the career progress and research productivity of a sample of grant applicants. The purpose is to assess the respondent's satisfaction with the grants application and award process and to get their opinions on specific modifications NIH is considering for improving policies and procedures.

SAMHSA's priority of evaluating its Knowledge, Development, and Application (KDA) grants is also a good example of HHS's commitment to continuous improvement of programs. Evaluating the current KDA grants will generate new data on the effectiveness of alternative models of managed care for treatment and prevention of substance abuse and mental disorders, including specific programs for homelessness among adults with serious mental illnesses and brief interventions for marijuana dependence.


1. FDA programs are principally authorized by legislation other than the PHS Act, specifically the authority of the Agriculture, Rural Development, Food and Drug Administration and Related Agencies Appropriations Act. IHS programs are authorized under the Indian Health Care Improvement Act and the Indian Self-Determination Act; they are appropriated under the Department of the Interior and Related Agencies Appropriations.