Performance Improvement 1997. Substance Abuse and Mental Health Services Administration


MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services for substance abuse and mental illness.

Evaluation Program

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its overall programs and individual grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care; the accountability of Federal funds; and the achievement of SAMHSA's programmatic and policy objectives.

To the greatest extent appropriate and feasible, SAMHSA will encourage the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.

SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: (1) services and (2) knowledge development and application (KDA). The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA will evaluate each of its service programs and provide information to program managers about the accountability of Federal funds. In addition, the evaluations of KDA programs will generate new knowledge to lead the field in the development of policies that improve services. Program and evaluation staff must work together to identify clearly the questions or goals each grant program will address and to propose appropriate evaluation strategies.

The two types of grant programs (service delivery and KDA) represent the two facets of SAMHSA's mission. SAMHSA's leadership in the field depends on the successful interaction of these two facets of its mission. Through evaluation, SAMHSA must identify effective approaches to prevention, treatment, and rehabilitation. Through service delivery funds, SAMHSA must provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are to develop, identify, and disseminate effective strategies and systems for prevention, treatment, and rehabilitation.

SAMHSA is implementing an integrated model of evaluation and planning. Strategic planning identifies priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is attempting to improve performance by identifying performance goals associated with its strategic plan. The formulation of programmatic and evaluation priorities includes consultation with the SAMHSA and Center Advisory Councils and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that can be evaluated. Evaluations will show how well the overall grant programs have achieved their objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA's priorities and objectives.

This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way SAMHSA can best serve its customers by enhancing the quality of public substance abuse and mental health services.

In compliance with Public Health Service (PHS) guidelines for the technical review of evaluations, SAMHSA has established a standing committee of PHS staff who are evaluation specialists. Representatives of the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members of the committee. The SAMHSA evaluation officer is the committee chair. The committee does not generally review the evaluation proposals of individual grantees. It does review proposals for broader, more comprehensive evaluations, such as the cross-project evaluations of grant programs.

Evaluation project proposals are generally prepared by SAMHSA program staff in the various Centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives and research questions; appropriateness and feasibility of the specifications for evaluation design and methods; appropriateness of the plans for dissemination of results; and use of previous relevant evaluations and existing program data systems. Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policies, priorities, and evaluation program.

Summary of Fiscal Year 1996 Evaluations

In FY 1996, SAMHSA completed a report entitled Overview of Addiction Treatment Effectiveness. This document presents an overview of the research literature substantiating the effectiveness of addiction treatment. The report describes specific treatment approaches, settings, components, and services and provides an analysis of the effectiveness of each. It was prepared by conducting comprehensive literature searches, reviewing meta-analyses, examining literature reviews, and reviewing several hundred research articles.

Also during FY 1996, SAMHSA completed a report entitled Employment Outcomes of Indigent Clients Receiving Alcohol and Drug Treatment in Washington State. This report was prepared by researchers at the University of Washington and the Washington State Division of Alcohol and Substance Abuse. It reports results from a project that evaluated substance abuse treatment outcomes for indigent clients served in Washington State. The treatment group included 499 individuals in various treatment modalities. The comparison group included 168 clients who were assessed but did not initiate treatment. Analyses were conducted on employment outcomes, using data obtained from official State employment records. The employment data cover a 12-month period before treatment and an 18-month period after treatment.

The study found that the duration of treatment is an important predictor of employment outcomes. Clients staying in treatment longer experience better outcomes, as do clients who complete treatment. A second finding was that the benefits of treatment, in enhanced earnings, compare favorably with the costs of treatment.

The policy implication of the study is that less treatment may not necessarily be less costly in the long run. Limiting coverage for substance abuse treatment for public clients will reduce short-run treatment costs, but may compromise important benefits of treatment, such as enhanced employment and increased earnings.

Evaluations in Progress

SAMHSA currently has several major evaluations under way in the following general areas: program accountability, knowledge development and application, and managed care.

Program Accountability

SAMHSA conducts evaluations for program accountability in compliance with the GPRA. These evaluations are undertaken to inform program management and help managers refine program operations. This is the primary type of evaluation conducted on SAMHSA's service grant programs. For example, the children's mental health service program has an ongoing evaluation. The evaluation will yield continuous information on program implementation and on outcomes for children and families served. This information will be used for reports to Congress, feedback to grantees, program development, and performance improvement.

Knowledge Development and Application

The primary purpose of SAMHSA's KDA program is to generate new knowledge to lead the field in the development of policies that improve substance abuse and mental health services. For example, evaluation results on substance abuse prevention for high risk youth will allow policymakers to draw inferences about the effectiveness of certain interventions for this population. An evaluation of the program for Access to Community Care and Effective Services and Supports will provide information for the design of ongoing service programs at the Federal, State, and local levels.

The National Treatment Improvement Evaluation Study is a cross-site evaluation study examining the effectiveness of demonstration grants funded by the Center for Substance Abuse Treatment. The purpose of the study is to assess the extent to which treatment enhancements improve substance abuse treatment outcomes over time. Preliminary results from this study provide strong support for the efficacy of substance abuse treatment in reducing levels of substance use, reducing involvement in criminal activities, and increasing employment. The final report will be available in FY 1997.

Managed Care

The Center for Substance Abuse Treatment has initiated an important managed care evaluation to assess the impact of State managed care initiatives on substance abuse treatment in terms of access, cost, and quality. Currently, there is little information at the State, provider, or client levels on the impact of managed care on the provision and outcomes of substance abuse treatment services. Several States will be selected for rigorous, statistical evaluation, which will include measures of costs, access, quality, and treatment outcomes. Results will be used to improve State substance abuse managed care programs and for Federal policy making related to managed care and health care reform.

New Directions for Evaluation

SAMHSA is designing its evaluation activities to guide programmatic and policy decisions. The evaluation activities will complement the SAMHSA and HHS strategic plans and will respond to emerging trends such as managed care.

One SAMHSA evaluation priority is knowledge development and application. SAMHSA's KDA program is designed to answer specific, important, policy-relevant questions. These questions will be designed to provide critical information to improve the Nation's mental health and substance abuse treatment and prevention services. Evaluations of the current KDA program will generate knowledge on such topics as the relative effectiveness of alternative models of managed care for treatment and prevention of substance abuse and mental disorders, the relative effectiveness of alternate models for preventing homelessness among adults with serious mental illnesses, and the efficacy of a brief intervention for marijuana dependence.

Another of SAMHSA's evaluation priorities is to assess the impact of managed care on the availability of services to populations in need, and to develop standards for measuring quality and outcome. SAMHSA's managed care evaluation strategy is designed to leverage important knowledge that directly affects the agency mission. This knowledge will be acquired through focused applied health services research, evaluations, demonstrations, and epidemiological and service capacity studies.