Performance Improvement 2009. Chapter I - Assessing Program Effectiveness

01/01/2009

Managing a Big Department

The Department of Health and Human Services (HHS) administers 337 programs (listed in Appendix A; also, see descriptions of all Federal programs in the Catalog of Federal Domestic Assistance, www.cfda.gov). The HHS budget included $721 billion for fiscal year (FY) 2008, the period covered by this report. Of this amount, Congress directed more than $800 million for evaluation and related activities through the set-aside provision of the Public Health Service Act (Section 241) which allows the Secretary of HHS to use a portion of the amounts appropriated for programs authorized under the Act for the evaluation, directly, or by grants and contracts, of the implementation and effectiveness of these programs[1]. Additionally, Congress annually appropriates significant other funds for both directed and discretionary evaluations. Since these make up part of the essential evaluation activities of the Department, they too are included in this report.

Why We Evaluate

Successful programs rely on effective evaluation. Thoughtful program evaluation can improve the delivery of public services and ensure that programs are efficient, targeted to their intended clients, and well managed. Important questions addressed by program evaluations include: what are the circumstances in which programs exist, who are the people that need services, and which program activities work best? The initial needs that give rise to programs, the knowledge base on which they are constructed, and the environment in which they operate, are not static. Actions by many individuals and demographic, economic, political, and social conditions are in flux. Legislators, policy makers, and managers must stay abreast of change. Congress and the Executive branch are responsible for the design, modification and implementation of laws and rules governing programs and both branches of government need the results of ongoing research, evaluation, and policy studies to effectively monitor and appropriately modify programs.

Funding Evaluations

Under the Public Health Service Act provision, about 2.4 percent of funds annually appropriated under the Act are used both to conduct evaluations and to fund other activities identified by Congress. The Public Health Service Act requires that the Secretary report annually, to the Senate Health, Labor and Pensions Committee and the House Energy and Commerce Committee, the findings of evaluations conducted under these provisions. As a courtesy to Congress, though not required by the Public Health Service Act, evaluations funded pursuant to other yearly administrative appropriations and statutory administrative authority are also included. This report, the 15th in this series, provides summaries of recently completed studies funded with both set-aside as well as other program evaluation funds.

 

What Do We Evaluate?

Evaluation is an essential means of achieving outstanding program performance. Evaluations begin with questions for which we seek answers. Without questions, there would be no need to study how well programs perform, what they accomplish, and what the circumstances are in which they operate. Study results emerge as significant signposts along the path of program review and improvement efforts. Evaluation implies critical judgment and reaching conclusions about merit and value. This year’s collection of completed studies includes a wide range of significant findings that are potentially of broad interest.

Evaluation priorities respond to and are guided by Congressional oversight, Executive branch decision-making, program management needs and performance measurement systems, including Presidential guidance and the Government Performance and Reports Act of 1993 (GPRA). During the past few years, executive agencies have also incorporated challenges and measurement involved in the Performance Assessment Reporting Tool (PART) issued by the Office of Management and Budget. Evaluation activities must also respond to changing realities in the programs themselves and in the environments in which the programs operate as well as in response to advice and recommendations derived from earlier evaluations.

Collaboration and Participation

To continue to refine and strengthen programs administered by HHS and the way they are evaluated reader feedback is welcomed. For this purpose, a one-page evaluation form is included at the end of the report. Please submit comments or recommendations, by either mail or email.

As indicated above also, for interested readers of this report, you may also see the entire database of over 8,000 abstracts and summaries, many with links to the original reports, as well as obtain access to the reports in this Performance Improvement series online: http://aspe.hhs.gov/pic/performance.

 

Performance Measurement Improvement Context

 

There are many systems of accountability to help assure effective and efficient functioning of government programs. Some are internal the HHS, others operate across all Executive Branch agencies. Congressional oversight provides performance review, and the public is the final judge of performance. Internally within HHS, the work of each program and agency (see Appendix C) is subject to review and approval by the Office of the Secretary, reflecting the advice gathered from several Assistant Secretaries.

As with most Departments and many individual agencies, HHS has a large Office of the Inspector General (OIG) which annually sets an ambitious study agenda, not only to audit finances of certain programs, but to examine how well programs are meeting their statutory and social objectives. Every OIG, including ours, functions under a dual reporting authority. While the OIG’s budget, staffing, and organizational functioning is subject to oversight by HHS, the OIG is also authorized and required to provide the results of its audit, program evaluation, and inspection to the Congress, uncensored by HHS policy and political leadership, thus assuring that its voice is heard on matters of public urgency.

Across the entire Executive Branch, during the annual budget development and review process, run by the Office of Management and Budget (OMB) on behalf of the President, the success and failure of programs of government is reviewed and correspondingly appropriate future resource levels judged.

Outside the executive branch, the most fundamental and far reaching control and performance assessment results from the work of the Congress through its oversight committee function. The Congress makes the laws, and it can likewise change them, and does, as a result of reassessments regarding the adequacy and performance of programs. Various staff offices support the work of Congress, including, notably, the Government Accountability Office (GAO). The GAO, used to focus on financial audit and accounting. It now engages in a wide array of performance studies at both a micro (individual program) level and macro (broad or cross-cutting governmental function) levels.

Most importantly, informed, engaged public groups and individuals regularly influence the operation and direction of programs, most frequently through the direct actions of elected and appointed officials. Civil servants, providing institutional memory and continuity, act to assist in designing and implementing the wishes of the public and elected and appointed leaders. Non-governmental organizations (NGOs) and active media can and frequently do provide a form of oversight, assessment, and voice for improving both the underlying law and the functioning of programs. These groups help to educate the public about what actions are being taken on their behalf, thereby better enabling an informed citizenry to more fully participate in its government.

 

How Evaluations Originate

Some evaluations are required by statute, others are considered essential by the President, and the Department, or an individual agency. Evaluation completes other core Federal management responsibilities: strategic planning, policy and budget development, and program operations (Figure 1).

Figure 1: Federal management responsibilities: strategic planning, policy and budget development, and program operationsD

In FY 2008, the HHS budget included $711 billion for HHS programs. Of this amount, Congress directed more than $800 million for evaluation and related activities through the Public Health Service Act Set Aside provision [Section 241(a) of the Act]. Successful evaluation increases the likelihood of effective delivery of public services through these programs and insures that programs are efficient, targeted to their intended clients, and well managed. Additional funds, through general and directed authorities, are also available for research, demonstrations, and evaluations by agencies of HHS.

 

Role of Evaluation

Programs need to provide good results for the individuals served, use resources wisely, and achieve the goals intended by Congress and the President. This obligatory report to Congress on Performance Improvement continues the effort to provide a strategic and analytic presentation of evaluations. Many provisions of public laws and executive orders include the need for evaluations and systematic review of programs and goals across the department.

This report reflects the efforts of departmental staff to measure, test, and evaluate the effect of programs, and to provide information that enables managers and policy makers to address changes that may be needed in existing programs or in revising policies, regulations and statutory provisions.

HHS evaluations directly support several efforts. Evaluations help government officials and members of the Congress make decisions related to programs, policies, budgets, and strategic planning. Evaluations enable managers to improve their program operations and performance. In addition, these results and methodologies are useful to the larger health and human services community of state and local officials, researchers, advocates, and practitioners to improve the performance of their programs.

Types of Evaluation

Classic Evaluation

Traditional program evaluation categories include process/implementation, experimental impact, non-experimental (or quasi-experimental), cost-benefit analysis, and other outcome studies.

  • A cost-benefit analysis, examining the advantages and costs of one or more program designs, could be carried out before a program has been implemented.
  • During the first several months of a program’s existence, before there are discernable outcomes to measure, a process or implementation evaluation could be carried out to see if the program is being set up as intended.
  • Fully experimental evaluations, or random-assignment studies, are considered the gold-standard of evaluation because they include both program and control groups so the results of the program can be compared to a group intended to be identical in every way except for the role of the program being tested.
  • Finally, non-experimental or quasi-experimental studies seek to find natural circumstances that mimic to some extent what is created artificially by fully experimental studies so that comparisons can be drawn.

Performance measurement differs somewhat from and can fully complement evaluations. While performance measurement may use some of the same types of evaluative tools, the goal is more directed. While an evaluation will typically test a hypothesis, performance measurement must start with the goal of measuring observed performance against particular expectations or criteria for success.

Type by Use

A second way of thinking about types of evaluations is to examine how the information is intended to be used. HHS evaluations assess performance (efficiency, effectiveness, and responsiveness) of programs or strategies, the use of information in strategic planning, programmatic or policy oriented decision-making, and program improvement. Evaluations serve one or more of these objectives (Figure 2):

  • Improve Performance Measurement  — Monitor annual progress in achieving departmental strategic and performance goals. HHS invests in evaluation funds to develop and improve performance measurement systems and the quality of the data that support those systems. The emphasis during development, implementation, and refinement of programs is on results and specific measurements as required under the Government Performance and Results Act.
  • Strengthen Program Management and Development  — Address the need of program managers to obtain information or data that will help them effectively design and manage programs more efficiently and ensure successful results. Focus on developmental or operational aspects of program activities and provide understanding of services delivered and populations served.
  • Assess Environmental Factors  — Seek to understand the forces of change in the health and human services environment that influence the success of our programs. Such understanding allows us to adjust our strategies and continue to deliver effective health and human services.
  • Enhance Program Effectiveness and Support Policy Analysis  — Determine the impact of HHS programs on achieving intended goals and objectives and examine the impact of alternative policies on the future direction of HHS programs or services.

Figure 2: Types of evaluations is to examine how the information is intended to be used.D

Evaluation Resources

Evaluation activities of HHS agencies and offices are supported with both general program funding and with a portion of the funds appropriated under the Public Health Service Act “ set-aside” authority.

General Program Funding

Program managers, operating under either discretionary or directed authority may use program funds to support contracts to design and carry out evaluations and analyze evaluation data. In some cases, a program’s statutory authority calls for specially mandated evaluations, and program funds are used directly to support these studies. Agencies for which one or both examples of such funding apply include the Administration for Children and Families (ACF) and the Centers for Medicare & Medicaid Services (CMS).[2] Such funds for evaluation are also available for the Administration on Aging.[3]
 

Public Health Service Act Set-Aside Authority

The Public Health Service Act, Section 241 set-aside authority was originally established in 1970, when the Congress amended the Act to permit the HHS Secretary to use up to 1 percent of appropriated funds to evaluate authorized programs. Section 241 limited the base from which funds could be reserved for evaluations to programs authorized by the PHS Act. Excluded were funds appropriated for the Food and Drug Administration, the Indian Health Service, and certain other programs that were managed by PHS agencies but not authorized by the Act (e.g., HRSA’s

Maternal and Child Health Block Grant and CDC’s National Institute for Occupational Safety and Health).[4]

The Consolidated Appropriations Act, 2008, authorized the Secretary to use up to 2.4 percent of the amounts appropriated for programs authorized by the Public Health Service Act for the evaluation of these programs. For Fiscal Year 2008, the year reflected in the studies here reported, agencies were budgeted a total of $913 million from the set-aside authority:

Administration for Children and Families (ACF) -- $11 million

Agency for Healthcare Research and Quality (AHRQ) -- $335 million

Centers for Disease Control and Prevention (CDC) -- $327 million

Health Resources and Services Administration (HRSA) -- $28 million

National Institutes of Health (NIH) -- $22 million

Substance Abuse and Mental Health Services Administration (SAMHSA) -- $123 million

Three staff components in the Office of the Secretary received a total of $47 million, shared between the Office of the Assistant Secretary for Planning and Evaluation (ASPE), the Office of Public Health and Science (OPHS), and the Office of the Assistant Secretary for Financial Resources (ASFR). In addition, the Office of the National Coordinator for Health Information Technology (ONC) received $19 million and the Office of the Assistant Secretary for Preparedness and Response (ASPR) received $3 million.

Substantial portions of the above funds are congressionally directed to pay for both general operating expenses and broad research activities.[5]

Most evaluations are started in one budget year, carried out in one or more subsequent years, and final reports, marking the completion of each study, may be delivered and available for the public in a third or subsequent year. Therefore, not all funds for studies completed in a particular year equate to the funds appropriated for that year.

Evaluation Management

Management of evaluations carried out by HHS agencies and offices involves: (1) planning and coordination, (2) project oversight, (3) quality assurance, and (4) dissemination of results (Figure 3). A description of each function follows.

Figure 3: Management of evaluations carried out by HHS agencies and offices involves: (1) planning and coordination, (2) project oversight, (3) quality assurance, and (4) dissemination of resultsD

Evaluation Planning and Coordination

The Government Performance and Results Act of 1993 (GPRA) requires that the Department establish a new five-year strategic plan every three years. The most recent was prepared last year for 2007-2012. This statute forms an essential basis for evaluation planning. HHS agencies and staff offices develop evaluation plans annually in concert with HHS program planning, legislative development, and budgeting cycles. Each agency or office evaluation plan generally states the evaluation priorities or projects under consideration for implementation. Typically, HHS evaluation priorities include: congressionally-mandated program evaluations, evaluations of Secretarial program or policy initiatives, assessments of new programs and ones that are candidates for reauthorization, and evaluations that support program performance management and accountability.

HHS evaluation planning activities are coordinated with three department-wide planning initiatives. First, HHS evaluation activities support the Department’s strategic planning and performance management activities in several ways. Completed evaluations are used in shaping specific HHS strategic goals and objectives. Evaluation findings provide important sources of information and evidence about the success of various HHS programs or policies. The HHS Strategic Plan highlights evaluations that document efficacy or effectiveness of strategic programs or policies and lists future evaluations that will benefit strategic planning. HHS agencies use findings from their evaluations to support GPRA annual performance reporting to Congress and program budget justifications.

Then, as specified in statute, the Secretary reports to the Congress the plans for using PHS evaluation set-aside funds before implementing these plans.

Project Oversight

HHS agencies and staff offices execute annual evaluation plans that involve developing evaluation contracts and disseminating and applying evaluation results. Where their subject matters relate, agencies seek to coordinate with one another.

The OIG performs independent evaluations through its Office of Evaluation and Inspections (OEI). OEI’s mission is to improve HHS programs by conducting inspections that provide timely, useful, and reliable information and advice to decision makers. Findings of deficiencies or vulnerabilities and recommendations for corrective action are usually disseminated through inspection reports issued by the Inspector General.

Quality Assurance and Improvement

Most evaluation projects are developed at the program or office level. A committee of agency- or office-level policy and planning staff members may conduct an initial quality review. Before a project is approved, a second committee reviews it for technical quality with expertise in evaluation methodology. Technical review committees generally follow a set of criteria for quality evaluation practice established by each agency. ASPE, for example, has a peer review committee that serves to improve the technical merits of ASPE proposals before final approval. Some HHS agencies have external evaluation review committees composed of evaluation experts from universities and research centers.

Since HHS began reporting to Congress in 1995 on completed evaluations through the Performance Improvement report series, the Department has focused attention on improving the quality of evaluations performed. In the past, Evaluation Review Panels, convened periodically, have contributed insights to HHS evaluation officers on the strengths and challenges of ensuring quality evaluations.

Dissemination of Evaluation Reports

Maintaining online electronic report libraries and distributing information on evaluation results is an important component of HHS evaluation management. The Department’s information and reports on major evaluations are available through the Web site of the HHS Policy Information Center (PIC), located at: http://aspe.hhs.gov/pic/performance (Appendix F contains additional information about how to access this information). ASPE’s PIC Web site offers users an opportunity to search  – by key word, selected program, or policy topics – the departmental evaluation report database and electronic report library maintained by ASPE. New entries in the online database focus on effective and clear summaries answering the basic questions: what was the study, why was it conducted, and what was learned. Through the online database, information regarding the work of evaluation within the department can be made known, speeding the dissemination of important factual information regarding work of the Department.

Additionally, the results of HHS evaluations are disseminated on agency and office websites through targeted distribution of printed reports, and research briefs as well as presentations at professional meetings and conferences. HHS researchers also participate in the broader research community through articles in specialist publications and refereed journals.

View full report

Preview
Download

"PerformanceImprovement2009.pdf" (pdf, 1.26Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®