Performance Improvement 1998. Substance Abuse and Mental Health Services Administration


MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services for substance abuse and mental illness.

Evaluation Program

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its overall programs and individual grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care; the accountability of Federal funds; and the achievement of SAMHSA's programmatic and policy objectives.

To the greatest extent appropriate and feasible, SAMHSA encourages the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.

SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: (1) services and (2) knowledge development and application (KDA). The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA evaluates each of its service programs so as to provide information to program managers about the accountability of Federal funds. In addition, the evaluations of KDA programs will generate new knowledge to lead the field in the development of policies that improve services.

The two types of grant programs (service delivery and KDA) represent the two facets of SAMHSA's mission. SAMHSA's leadership in the field depends on the successful interaction of these two facets. Through KDAs, SAMHSA must identify effective approaches to prevention, treatment, and rehabilitation. Through service delivery funds, SAMHSA must provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are to develop, identify, and disseminate effective strategies and systems for prevention, treatment, and rehabilitation.

SAMHSA is implementing an integrated model of evaluation and planning. Strategic planning identifies priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is improving performance management by identifying annual performance objectives and measures. The formulation of programmatic and evaluation priorities includes consultation with SAMHSA, Center Advisory Councils, and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that may be evaluated. Evaluations will demonstrate the extent to which the grant programs have achieved their overall objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA's priorities and performance objectives.

This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way, SAMHSA can best serve its customers by enhancing the quality of publicly-funded substance abuse and mental health services.

In compliance with the Public Health Service (PHS) guidelines for the technical review of evaluations, SAMHSA has established a standing committee of PHS staff who are evaluation specialists. Representatives of the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members of the committee. The SAMHSA evaluation officer is the committee chair. The committee does not generally review the evaluation proposals of individual grantees; rather, it reviews proposals for broader, more comprehensive evaluations, such as the cross-project evaluations of grant programs.

Evaluation project proposals are generally prepared by SAMHSA program staff in the various Centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives and research questions, appropriateness and feasibility of the specifications for evaluation design and methods, appropriateness of the plans for dissemination of results, and use of previous relevant evaluations and existing program data systems. Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policies, priorities, and evaluation program.

Summary of Fiscal Year 1997 Evaluations

During FY 1997, SAMHSA completed six evaluations. A summary of the resulting reports follows.

The NTIES: National Treatment Improvement Evaluation Study: Final Report (5346.1), highlighted in Chapter II, was a comprehensive assessment of the results of 157 three-year demonstration grants and cooperative agreements funded by the Center for Substance Abuse Treatment (CSAT) beginning in 1990-91. These grants were intended to improve alcohol and drug treatment in selected target cities with severe substance abuse problems, among critical populations such as minority groups and women, and among persons in the criminal justice system. Based on a final sample of 6,600 clients who entered treatment, the results showed a substantial reduction in the selling of drugs and the commission of violent crimes, high-risk sexual behaviors related to HIV transmission, medical visits, homelessness, and inpatient mental health visits. Results also showed a substantial increase in employment.

The CSAT Demonstration: Evaluation of Job Corps Drug Treatment Enrichment Program (4523) was a 4-year demonstration project sponsored by CSAT and put in place within the Department of Labor's Job Corps program. The Drug Treatment Enrichment Program (DTEP) was implemented in four experimental Job Corps Centers. These were matched with four control centers that provided the standard Job Corps Alcohol and Other Drugs of Abuse program. The evaluation found that DTEP students reduced marijuana and crack or cocaine use following their Job Corps experience significantly more than did the students in the control groups. DTEP students were less likely to engage in moderate or extensive use of marijuana or to use other drugs, but no differences were found for alcohol abuse. DTEP students were less likely to report selling or helping to sell drugs after Job Corps. No differences between DTEP and controls were found for other criminal activities. DTEP improved students' mental health. DTEP students with mental health problems experienced a significantly higher job placement rate after Job Corps than did control students, although the program had no effect on job placement rates of students without mental health problems. DTEP appeared to have no significant positive effect on duration of stay in Job Corps or on overall educational and vocational gains. The final report makes recommendations regarding the integration of DTEP into Job Corps and other related policy implications.

The Center for Substance Abuse Prevention (CSAP) released the findings of a demonstration evaluation, Evaluation of Model Projects for Pregnant and Postpartum Women and Their Infants (5674). The evaluation assessed the effectiveness of the demonstration program in enhancing services coordination and in increasing the availability and accessibility of substance abuse prevention services delivered to clients. The study also documented the effectiveness of program models in decreasing alcohol and drug use among substance-abusing women, and enhancing the healthy development of their children. Ten projects funded by CSAP in 1992 were selected to participate in the evaluation. Data collected by the grantees were used for cross-site analyses. The major evaluation finding was that from intake to delivery, the treatment group members reported significantly lower use of marijuana, crack, and other illicit substances than comparison group members. These differences, however, were not significant by the 6-month postpartum followup assessment. Overall, treatment and comparison group members gave birth to infants with similar birth-related outcome characteristics. However, when the history of the mother's use of crack was taken into account, treatment babies had significantly higher birth weights than the babies in the comparison group.

CSAP also conducted a Report on the CTS National Evaluation (4514.1). The training system included the development and delivery of more than 30 curricula to more than 10,000 participants. Recipients included community teams, health professionals, State administrators, and volunteers engaged in prevention activities. The report found that the training system was innovative in several important respects. It focused on community coalitions and associations of health professionals that are responsible for planning and delivering prevention services, rather than concentrating on target populations. Curricula were developed quickly in response to emerging issues in the field. The training system had a continuing commitment to quality control and continual curricular improvement.

CSAP has completed Findings from the Evaluation of the Faculty Development Program, Volume I: Final Report (4510). This program offers training to faculty members from schools of medicine, nursing, and social work, and from graduate psychology programs. The purpose of the development program was to enhance research, teaching, and clinical practices on issues of alcohol, tobacco, and other drugs (ATOD). The evaluation found that faculty fellows, after participating in the program, reported (1) spending significantly more time on ATOD-related activities, (2) conducting more ATOD research, (3) incorporating more ATOD information in courses, (4) joining ATOD professional organizations, (5) increasing ATOD teaching or clinical skills, (6) producing more ATOD publications, and (7) making more ATOD presentations at conferences and other events than they had prior to their participation in the program.

The Center for Mental Health Services (CMHS) produced Final Report: Review of the CMHS Mental Health Care Provider Education in HIV/AIDS (6695). The program supports the provision of state-of-the-art training targeted to (1) traditional mental health care providers (psychiatrists, psychologists, nurses, and social workers); (2) other first-line providers of mental health care services (primary care physicians and medical students); and (3) nontraditional providers of such services (clergy or alternative health care workers). The goal of the program was to improve access to appropriate and adequate mental health services for people living with HIV/AIDS and for their families and partners. The evaluation found that training participants increased their knowledge about the mental health aspects of the disease, their willingness to serve HIV-infected individuals, and their confidence that they could do so effectively. The recommendations of the report stressed the need for training in the mental health aspects of HIV/AIDS throughout the country, especially as affected areas and populations change over time.

Evaluations in Progress

SAMHSA currently has major evaluation projects under way in two areas. First, CSAT's National Evaluation Database Service (5994) is providing centralized data management and analysis for the evaluation of several large demonstrations targeted to special populations, including female, adolescent, criminal justice, culturally distinct, and rural populations. The service represents part of an evaluation strategy that builds upon prior findings, and which seeks to identify a set of consistent evaluation questions that apply across similar substance abuse treatment programs targeted to special populations. The service has identified data elements to provide uniform information across sites so that comparisons of effectiveness could be made.

Second, CSAP is conducting a National Evaluation of the Community Partnership Demonstration Project (4997). This project was designed to identify successful prevention and partnership strategies and common inhibitors to forming successful partnerships and prevention programs. This impact evaluation has focused on assessments of short-term outcomes specific to each interview site, long-term outcomes reliant upon uniform surveys, and community-wide indicators of alcohol and drug abuse across several sites. Data have been collected through a questionnaire administered in schools and a telephone survey for adults.

Other evaluations in progress are looking at services integration for homeless persons with chronic mental illness (4980.1), effects of managed care on substance abuse treatment outcomes (6454), evaluation of substance abuse treatment improvement protocols (6737), long-term effects of drug abuse treatment (6738), and an evaluation of opioid treatment program accreditation (6739).