Performance Improvement 2007. Chapter III - Program Evaluation at the Department of Health & Human Services

01/01/2007

 

The Department of Health and Human Services (HHS) funds or conducts many evaluations some required by statute, others considered essential by an individual agency, the Department, or the President.  Evaluation is a core Federal management responsibility with strategic planning, policy and budget development, and operating programs (Figure 1, Performance Management System). 
 

 

Performance Management System

Chart showing Performance Management System where Delivering Efficient, High Quality Program Performance is the nexus of four factors: Evaluation and Assessment, Strategic Planning, Policy Formulation and Budgeting, and Program Operations

 

As currently listed in the Catalog of Federal Domestic Assistance (www.cfda.gov), the Department is responsible for more than 330 separate programs.  In FY 2006, HHS spent $613 billion.  Of this amount, $2 billion was for demonstration and evaluation activities, including $832 million provided through the Public Health Service Act Set Aside provision [Section 241(a) of the Act]. Successful evaluation increases the likelihood of effective delivery of public services through these programs and insures that programs are efficient, targeted to their intended clients, and well managed.

 

Role of Evaluation

Programs need to provide good results for the individuals served, spend tax dollars wisely, and achieve the goals intended by Congress and the President. This obligatory report to Congress on Performance Improvement continues the effort to provide a strategic and analytic presentation of evaluation studies.  With the implementation of a unified Strategic Plan, as required by the Government Performance and Results Act of 1993, the Department recognizes its responsibility  to evaluate programs and to assure that evaluation funds are targeted to address the core goals and objectives of both the Congress and Executive branch.  This report reflects the important role evaluations have to test, weigh, measure and judge the success of management performance, program outputs, and social outcomes.

 

Evaluations are essential to successful achievement of the HHS mission. Assessing various aspects of agency functioning allows staff to identify means of improving individual program performance.  HHS evaluations directly support several efforts. Evaluations help government officials and members of the Congress make decisions related to programs, policies, budgets, and strategic planning. Evaluations enable managers to improve their program operations and performance. Evaluation results and methodological tools are useful to the larger health and human services community of state and local officials, researchers, advocates, and practitioners  to improve the performance of their programs.

 

Types of Evaluation

At their best, HHS evaluations assess the performance (efficiency, effectiveness, and responsiveness) of programs or strategies through the analysis of data or information collected systematically and ethically; the effective use of resulting information in strategic planning, program or policy decision-making and program improvement.  Evaluations serve one or more of the following objectives (Figure 2, Types of Evaluation):

 

Enhance Program Effectiveness and Support Policy Analysis — Determine the impact of HHS programs on achieving intended goals and objectives and examine the impact of alternative policies on the future direction of HHS programs or services.

 

Improve Performance Measurement — Monitor annual progress in achieving departmental strategic and performance goals.  We invest evaluation funds to develop and improve performance measurement systems and improve the quality of the data that support those systems.  Performance measurement is a high priority for HHS agencies.  The emphasis during development, implementation, and refinement of programs is on results and specific measurements are required under the Government Performance and Results Act.

 

Assess Environmental Factors — Seek to understand the forces of change in the health and human services environment that influence the success of our programs.  Such understanding allows us to adjust our strategies and continue to deliver effective health and human services.

 

Strengthen Program Management and Development — Address the need of program managers to obtain information or data that will help them effectively design and manage programs more efficiently and ensure successful results. Focus on developmental or operational aspects of program activities and provide understanding of services delivered and populations served.


 

Types of Evaluation

Figure labeled types of Evaluation, showing: Analyze Policy and program effectiveness (laws, regulations, guidelines); Measure Program Performance (functions, outputs, outcomes); Identify successful management practices (for Congressional, Executive and Agency leadership); and Assess External Environment (demographics, economics, physical threats)

 

Evaluation Resources Broadly

 

Evaluation activities of the various HHS agencies are supported with both general program funding and with some of the funds appropriated under the Public Health Service Act “set-aside” authority. Under the first, program managers, operating under discretionary authority, use program funds to support contracts to design and carry out evaluation studies and analyze evaluation data. In some cases, a program’s legislative authority calls for specially mandated evaluations, and program funds are used directly to support these studies.
 

Public Health Service Section Set-Aside Authority

 

The Public Health Services Section 241 set-aside authority was originally established in 1970, when the Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs.  Section 241 limited the base from which funds could be reserved for evaluations to programs authorized by the PHS Act.  Excluded were funds appropriated for the Food and Drug Administration, the Indian Health Service, and certain other programs that were managed by PHS agencies but not authorized by the Act (e.g., HRSA’s

Maternal and Child Health Block Grant and CDC’s National Institute for Occupational Safety and Health).[1]  In addition, programs may have other authorized sources of funding for evaluation activities.[2]  Section 207 of the Departments of Labor, Health and Human Services, and Education, and Related Agencies Appropriations Act, 2006, authorized the Secretary to use up to 2.4 percent of the amounts appropriated for programs authorized by the Public Health Service Act for the evaluation of these programs. 

For Fiscal Year 2006, the year reflected in the studies here reported, agencies were budgeted a total of $831 million from the set-aside authority, a portion of which was used for evaluation activities:

  • Administration for Children and Families (ACF) -- $11 million
  • Agency for Healthcare Research and Quality (AHRQ) -- $319 million
  • Centers for Disease Control and Prevention (CDC) -- $267 million
  • Health Resources and Services Administration (HRSA) -- $28 million
  • National Institutes of Health (NIH) -- $24 million
  • Substance Abuse and Mental Health Services Administration (SAMHSA) -- $121 million
  • Three staff components in the Office of the Secretary received a total of $40 million:
  • Office of the Assistant Secretary for Planning and Evaluation (ASPE)
  • Office of Public Health and Science (OPHS)
  • Office of the Assistant Secretary for Resources and Technology (ASRT)
  • In addition, the Office of the National Coordinator for Health Information Technology (ONC) received $19 million.

Most evaluation studies are started in one budget year, carried out in one or more subsequent years, and final reports, marking the completion of each study, may be delivered and available for the public, in a third or subsequent year. Therefore, the studies completed in a particular year cannot be equated to the funds appropriated for the same year.

This Performance Improvement 2007 report includes studies funded through the Public Health set-aside authority as well as studies supported with other appropriated funds.

Evaluation Management

Management of evaluations carried out by HHS agencies and offices and coordinated by ASPE involves: (1) planning and coordination, (2) project oversight, (3) quality assurance, and (4) dissemination of results (Figure 3, Evaluation Management Cycle).  A description of each function follows.

Figure 3: Evaluation Management Cycle

Chart with 6 subjectsin a one direction cycle. Evaluation planning and coordination lead to project oversight implementation and performance leads to research and data collection leads to quality assurance and analysis leads to results dissemination with programs policy and redesign leads to strategic planning leads back to evaluation etc.

 

Evaluation Planning and Coordination

HHS agencies, ASPE, the Office of the Inspector General (OIG), and several other offices develop evaluation plans annually in concert with HHS program planning, legislative development, and budgeting cycles. Plan development is coordinated by ASPE.  Each agency or office evaluation plan generally states the evaluation priorities or projects under consideration for implementation. Typically, HHS evaluation priorities include: congressionally-mandated program evaluations, evaluations of Secretarial program or policy initiatives, assessments of new programs and ones that are candidates for reauthorization, and evaluations that support program performance management and accountability.

HHS evaluation planning activities are coordinated with three department-wide planning initiatives. First, HHS evaluation activities support the Department’s strategic planning and performance management activities in several ways.  Completed evaluation studies are used in shaping specific HHS strategic goals and objectives.  Evaluation findings provide important sources of information and evidence about the success of various HHS programs or policies. The HHS Strategic Plan highlights evaluations that document efficacy or effectiveness of strategic programs or policies and lists future evaluations that will benefit strategic planning.  HHS agencies use findings from their evaluations to support the Government Performance and Results Act (GPRA) annual performance reporting to Congress and program budget justifications.

Second, Congress requests that HHS coordinate and report to Congress regarding all of its research, demonstration, and evaluation (RD&E) programs to ensure that the results of these projects address HHS program goals and objectives.  ASPE and the Assistant Secretary for Resources and Technology work together with HHS agencies to provide the Congress with a special annual research, demonstration, and evaluation budget plan that coincides with the preparation of the President’s fiscal year budget.  The plan outlines HHS agency research, demonstration, and evaluation priorities as related to the Department’s strategic goals and objectives (Figure 4, Evaluation Reporting Cycle).

 

Figure 4: Evaluation Reporting Cycle

Figure shows a large cycle of 5 entities with the Research, Demonstration and Evaluation (RD&E) Report feeding the Evaluation Budget and Plan Approved by OMB, Congress and the President. This approval is followed by the Set-Aside Funds and Evaluation Report which is followed by the conduct of evaluations. These evaluations are followed in turn by the Performance Improvement Report which cycles back to the RD&E Report. <br> The RD&E Report is at the top being fed by 6 arrows: Statutory requirements, Presidential and Secretarial Directives, the DHHS Strategic Plan, the Research Coordinating Council, the Agency Plan, and the OMB Performance and Assessment Reporting Tool (PART) advice.

 

Third, the Secretary report to the Congress his plan for using PHS evaluation set-aside funds before implementing the plan (Figure 4). 

 

Project Oversight

 

HHS agencies, ASPE, OASH, and the OIG execute annual evaluation plans that involve developing evaluation contracts and disseminating and applying evaluation results.  All agencies and their subunits (centers, institutes, and bureaus) coordinate with each other on research and evaluation project planning and release of final reports that relate to work of other HHS agencies.  While there is some oversight responsibility and execution capability in the Office of the Director or Administrator for each agency, the various agency subunits conduct much of the day-to-day evaluation activity.

 

The OIG performs independent evaluations through its Office of Evaluations and Inspections (OEI).  OEI’s mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decision makers.  Findings of deficiencies or vulnerabilities and recommendations for corrective action are usually disseminated through inspection reports issued by the Inspector General.  A summary of individual inspection reports and other OIG reports can be viewed on the World Wide Web at http://oig.hhs.gov/reports.html. OEI provides technical assistance to HHS agencies in conducting their evaluations.

 

Quality Assurance and Improvement

 

Most evaluation projects are developed at the program or office level.  A committee of agency- or office-level policy and planning staff members generally conducts the initial quality review.  Before a project is approved, a second committee reviews it for technical quality with expertise in evaluation methodology.  Technical review committees generally follow a set of criteria for quality evaluation practice established by each agency.  ASPE, for example, has a peer review committee that serves to improve the technical merits of ASPE proposals before final approval. Some HHS agencies have external evaluation review committees composed of evaluation experts from universities and research centers.

 

Since HHS began reporting to Congress in 1995 on completed evaluations through the Performance Improvement report series, the Department has focused attention on improving the quality of evaluation studies performed.  In the past, Evaluation Review Panels, convened periodically, have contributed insights to HHS evaluation officers on the strengths and challenges of ensuring quality evaluation studies.  HHS evaluation officers have had opportunities to discuss these strengths and challenges and identify steps to improve agency evaluation projects.

 

Dissemination of Evaluation Reports

 

Maintaining online electronic report libraries and distributing information on evaluation results is an important component of HHS evaluation management.  The Department’s information and reports on major evaluations are available through the Web site of the HHS Policy Information Center (PIC), located at: http://aspe.hhs.gov/pic/performance.  ASPE’s PIC Web site offers users an opportunity to search – by key word, selected program, or policy topics – the departmental evaluation report database and electronic report library maintained by ASPE. The PIC contains over 8,500 completed and in-progress evaluation and policy research studies conducted by the Department of Health and Human Services, as well as some studies completed outside of it by others.

 

During the past year, the Policy Information Center online database has been modified so that evaluation study project officers or other key agency staff now directly submit information online about current and project evaluations. This means that, as regards the online database, there is no delay in making information avalible to evaluation peers in other parts of the Department, and to the public at large. Researchers may now search to see what studies have been funded and are currently underway that may be relevant to their own research or planning activities. New entries in the online database are intended to focus on effective and clear summaries answering the basic questions: what was the study, why was it conducted, and what was learned. Through the online database, several months before annual reports are due to the Congress, much of the information regarding the work of evaluation underway can be known both to Congressional and Executive Branch staff and to the public as well, speeding the dissemination of important factual information regarding work of the Department. A positive result is reduced chances of duplication of efforts and speedier application of policy implications of evaluation work carried out.

 

Additionally, the results of HHS evaluations are disseminated through targeted distribution of final reports, articles in referenced journals, and presentations at professional meetings and conferences. Although individual HHS agencies have primary responsibility for disseminating results, ASPE continues its Department-wide efforts to expand dissemination of evaluation results to the larger research and practice communities through email lists, e-newsletters, and publications available on the PIC Web site.

 

View full report

Preview
Download

"PerformanceImprovement2007.pdf" (pdf, 717.63Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®