Performance Improvement 1998. Ensuring Effective Use of Evaluation Results


HHS is committed to ensuring that evaluations yield a high return on the investment of available program funds. Today, the Department's evaluation resources are used in several ways related to strategic planning, program, and policy development. Performance measurement and data systems are the primary mechanisms used to monitor progress in achieving departmental goals and objectives as well as specific program outcomes. Effectiveness of programs and strategies are in-depth evaluation studies to understand how HHS strategies and programs are linked to performance goals and objectives. Environmental assessments are prospective evaluation studies that assess how changes in the larger society affect the Department's programs and strategies. Program management and support evaluations are used to improve the management of health and social service programs and the quality of the Department's evaluation efforts. Each of these four uses, with examples of current HHS evaluations, is described below.

Performance Measurement and Data Systems

Implementation of the HHS strategic plan and the performance plans of the PHS agencies, pursuant to the Government Performance and Results Act (GPRA) of 1993, requires the Department to focus a portion of its evaluation activities on the development of performance objectives and measures and information systems necessary to produce the data needed to assess progress toward achieving its goals. During FY 1997, evaluation priorities of the individual HHS agencies included a number of projects serving this purpose. For example, ACF completed a project for Head Start Programs to develop outcome performance measures (e.g., how these children benefit from the program), in addition to process measures (e.g., how many teachers have the appropriate credentials?) (6693). FDA completed its fifth and final report on performance data for the Prescription Drug User Fee Act (PDUFA), documenting the improvements obtained in the last few years in speeding up the process for drug reviews and approvals (6079.2). ASPE produced a report synthesizing the state-of-the-art performance measurement for public health and developed recommendations for measures that could be used to monitor Federal-State performance partnership grants being considered for the future (6177). An example of a major survey providing nationally representative data is one completed last year by HRSA on the users of community health centers (CHC) and the services they were provided (5737). It produced, for the first time, nationally representative data on CHC users that is comparable to similar data on health care use from the National Health Interview Survey. The data are now being analyzed to address questions about users' race and ethnicity, health risk behaviors, most prevalent diagnoses, and services used.

Effectiveness of Programs and Strategies

Implementation of the Department's strategic and performance plans also requires evaluative information on how well the programs and strategies are working. Most of the HHS agency evaluations completed in FY 1997 provide this information. For example, HCFA submitted a report to the Congress on the quality of care in the Medicaid program (6302). It assessed the variations in the rate of performance of selected treatments and procedures on Medicaid beneficiaries for small areas within and between the States. Medical records in a sample of hospitals were also examined for three conditions: pediatric asthma, hysterectomy, and complicated delivery. Overall, the report concluded that care was considered adequate and comparable to that received by privately insured patients. Another example of a focused program effectiveness study is one conducted by HRSA on rural applications of telemedicine (5749). The project included a mail survey of all non-Federal hospitals located outside metropolitan areas in the summer of 1995, plus a short survey for hospitals doing only teleradiology and a longer survey for hospitals applying telemedicine to purposes beyond radiology. The study demonstrated that although rural telemedicine is in the earliest stages of development, it is expanding rapidly. HRSA is using the results to design a common telemedicine evaluation instrument for HRSA grantees and for consideration by other Federal sponsors, as well as in shaping the Rural Telemedicine Grant Program. In terms of evaluating research programs, NIH conducted an evaluation of the NIH Consensus Conference as an effective mechanism for translating research into practice (6284).

Environmental Assessment

The Department's strategic plan acknowledges that achievement of its goals and objectives is contingent on many external factors that are beyond the Department's control. For example, the managed care revolution is having a significant impact on access to services and health outcomes for those populations traditionally served by the public health system. Understanding the impact of these environmental forces on public health programs and the customers they serve is essential for assessing and adjusting the Department's goals and strategies in the future. A number of evaluations in FY 1997 addressed these environmental issues.

For example, AHCPR conducted a study to help clarify its future priorities in outcomes and effectiveness research and to provide insights regarding future opportunities for public-private partnerships in this area (6385). The study found that many private sector organizations conduct effectiveness research, often to monitor health care performance against accreditation standards. AHCPR will also use the findings to help guide its support to developing and validating effectiveness and outcome research tools used by the private sector. In another example, ASPE last year analyzed market barriers to the development of pharmacotherapies for substance abuse and addiction, particularly for abuse of and addiction to cocaine (6694). This report, featured in Chapter II, found several critical market barriers that must be taken into account in future HHS program and policy development. Since the substance abuse treatment market relies heavily upon State and Federal reimbursement, most substance abuse treatment services are subsumed under the mental health benefits of entitlement programs. Drug companies are reluctant to rely upon this kind of reimbursement in an age of shrinking budgets for mental health services.

Program Management and Support

Effective management of programs that achieve departmental goals and objectives is essential to the success of those programs. In FY 1997, the evaluations completed and in progress include a number of priorities devoted to assessing and improving the management of health and social service programs. For example, HCFA last year looked at the implementation of Ambulatory Patient Groups (APGs)--the Medicaid outpatient prospective payment system that groups patients for payment purposes rather than paying on a cost basis (6320). In this case study of Iowa's implementation of the APG system and an analysis of the reimbursement methodology, success was reported in reducing outpatient costs, where that was the immediate goal, and the system encouraged higher-cost facilities to reduce costs and reward lower-cost facilities. In another example of studies to improve program operations, OPHS conducted a Healthy People 2000 stakeholder study to help plan and prepare for the Healthy People 2010 goals and objectives (6491), also featured in Chapter II. This evaluation reviewed the successes and failures of the Healthy People 2000 benchmarks, with the hope of making this national framework of performance indicators on health status more results-oriented. The focus group participants recommended new communication avenues to make Healthy People 2010 available to more professionals and community leaders.