Performance Improvement 1996. Substance Abuse and Mental Health Services Administration


MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services for substance abuse and mental illness.

SAMHSA Evaluation Program

SAMHSA is committed to evaluating its overall programs and individual grant projects in order to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care; the accountability of Federal funds; and the achievement of SAMHSA's programmatic and policy objectives.

To the greatest extent appropriate and feasible, SAMHSA will encourage the use of comparable data elements and instruments across its evaluations in pursuit of a comprehensive evaluation system and to minimize respondent burden.

SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: services and demonstrations. The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA will evaluate each of its service programs to provide information to program managers about the accountability of Federal funds. In addition, the evaluations of demonstration programs will generate new knowledge to lead the field in developing policies that improve services. Program and evaluation staff must work together to identify clearly the questions or goals each grant program will address and to propose appropriate evaluation strategies.

The two types of grant programs represent two facets of SAMHSA's mission: service delivery and knowledge development. SAMHSA's leadership in the field is dependent upon the successful interaction of these two facets. Through evaluation, SAMHSA must identify effective approaches to prevention, treatment, and rehabilitation. Through service delivery funds, SAMHSA must provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are the development, identification, and dissemination of effective strategies and systems for prevention, treatment, and rehabilitation.

SAMHSA is now implementing a new integrated model of evaluation and planning. Strategic planning will identify priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act, SAMHSA is attempting to improve performance by identifying performance goals as part of its strategic planning process. The formulation of programmatic and evaluation priorities will include consultation with the SAMHSA and Center Advisory Councils, and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that can be evaluated. Evaluations will show how well the overall grant programs have achieved their objectives. SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use the results to refine SAMHSA's priorities and objectives.

This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way SAMHSA can best serve its customers by enhancing the quality of public substance abuse and mental health services.

In compliance with the PHS guidelines for the technical review of evaluations, SAMHSA has established a standing committee of PHS staff who are evaluation specialists. Representatives of the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members of the committee. The SAMHSA evaluation officer is the committee chair. The committee does not generally review the evaluation proposals of individual grantees. It does review proposals for broader, more comprehensive evaluations, such as the cross-project evaluations of grant programs.

Evaluation project proposals are generally prepared by SAMHSA program staff in the various centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives and research questions, appropriateness and feasibility of the specifications for evaluation design and methods, appropriateness of the plans for dissemination of results, and use of previous relevant evaluations and existing program data systems. Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policy and priorities and evaluation program.

Summary of Fiscal 1995 SAMHSA Evaluations

During fiscal 1995, SAMHSA completed several reports on its evaluations in the Center for Mental Health Services. Some of these studies focused on youth, one of the population groups that SAMHSA has identified as in greatest need of services. In particular, services for children and adolescents with serious emotional disabilities were the subject of a series of research studies funded by the Center for Mental Health Services (CMHS). The evaluations were highlighted in a special issue of the Journal of Emotional and Behavioral Disorders.

In one CMHS research study, investigators developed, implemented, and evaluated an intensive, adolescent-centered case management approach to treating homeless adolescents in Washington State. Results of a 3-month followup showed a significant decrease in symptoms of depression, problem behavior, and substance abuse; significant increases in self-esteem; and reports from the youth of an increase in quality of life. This study is significant because it is the first to formally assess the effectiveness of mental health-related services to homeless youths.

A second research study examined the effectiveness of an individualized, intensive case-managed approach to improving adjustment outcomes for foster children with emotional and behavioral disorders. Results of this community-based, case- controlled experiment indicate that children who received the additional individualized services showed significantly greater short-term improvements in some behavioral and emotional adjustments than did children who received standard practice foster care services.

Another CMHS-funded study assessed the impact of multisystemic family preservation therapy (MFP) on family functioning and problem behavior of delinquent adolescents in rural and urban South Carolina communities. MFP is a nonoffice-based therapy. Interventions are directed toward individuals and families, peer relations, school relationships, academic performance, and any other social system believed to be involved in the problem behaviors. In general, the MFP group demonstrated improvements in the amount of problem behavior, level of mothers' psychological distress, and family functioning following MFP treatment.

SAMHSA Evaluations in Progress

SAMHSA has 12 major evaluations under way in the following general areas: program accountability, evaluation of demonstrations, and managed care. A description of each type follows, with some examples.

Program Accountability

SAMHSA conducts evaluations for program accountability in compliance with GPRA. These evaluations are undertaken to inform program management and help managers refine program operations. This is the primary type of evaluation conducted on SAMHSA's service grant programs. For example, the children's mental health service program has an ongoing evaluation. The evaluation will yield continuous information on program implementation and on outcomes for children and families served. The information will be used for reports to Congress, feedback to grantees, program development, and performance improvement.

Evaluations of Demonstrations

Evaluations of demonstrations are designed to generate new knowledge for policy development. The primary purpose of SAMHSA's demonstration programs is to generate new knowledge to lead the field in developing policies that improve substance abuse and mental health services. For example, evaluation results on substance abuse prevention for high-risk youth will allow policymakers to draw inferences about the effectiveness of certain interventions for this population. The Job Corps evaluation, which will be completed in fiscal 1996, is assessing an enriched substance abuse treatment program for adolescents. An evaluation of the program for Access to Community Care and Effective Services and Supports (ACCESS) will provide information for the design of ongoing service programs at the Federal, State, and local levels.

The National Treatment Improvement Evaluation Study (NTIES) is a cross-site evaluation study examining the effectiveness of demonstration grants funded by the Center for Substance Abuse Treatment (CSAT). The study will assess the extent to which treatment enhancements improve substance abuse treatment outcomes over time. Preliminary results from NTIES provide strong support for the efficacy of substance abuse treatment in reducing levels of substance use, reducing involvement in criminal activities, and increasing employment. The final report will be available in fiscal 1997.

Managed Care

In fiscal 1996, CSAT is initiating an important managed care evaluation to assess the impact of the States' managed care initiatives on substance abuse treatment in terms of access, cost, and quality. Currently, there is little information at the State, provider, or client levels about the impact of managed care on the provision and outcomes of substance abuse treatment services. Five States will be selected for rigorous, statistical evaluation, which will include measures of costs, access, quality, and treatment outcomes. Results will be used to improve the States' substance abuse managed care programs and for Federal policymaking related to managed care and health care reform.

New Directions for SAMHSA Evaluation

SAMHSA is designing its evaluation activities to guide programmatic and policy decisions. The evaluation activities will complement the SAMHSA and HHS strategic plans and will respond to emerging trends such as managed care.

One SAMHSA evaluation priority is to evaluate demonstration programs and the impact of demonstration findings and knowledge- transfer activities on service delivery. In fiscal 1996, SAMHSA proposed a new demonstration program designed to answer specific, important policy-relevant questions. These questions will be designed to provide critical information to improve the Nation's mental health and substance abuse treatment and prevention services. Evaluations of the demonstrations planned for fiscal 1996 will generate knowledge on such topics as the relative effectiveness of alternative models of managed care for treatment and prevention of substance abuse and mental disorders, the relative effectiveness of alternate models for preventing homelessness among adults with serious mental illnesses, and the efficacy of a brief intervention for marijuana dependence.

Another of SAMHSA's evaluation priorities is to assess the impact of managed care on the availability of services to populations in need and to develop standards for measuring quality and outcome. SAMHSA's managed care evaluation strategy is designed to leverage important knowledge that directly impacts its mission. This knowledge will be acquired through focused applied health services research, evaluations, demonstrations, and epidemiological and service capacity studies. SAMHSA is currently considering an evaluation strategy that would first establish baseline information about the incidence, prevalence, functional disability, and negative consequences associated with mental health and substance abuse; characteristics of the service systems; and characteristics of individuals serviced by those systems. Evaluations would then assess the impact of changes in organization and financing services through managed care and health care system reforms.