MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services for substance abuse and mental illness.
SAMHSA Evaluation Program The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its overall programs and individual grant projects to assess--
- the effectiveness of treatment and prevention approaches and systems of care;
- the accountability for Federal funds; and
- the achievement of SAMHSA's programmatic and policy objectives.
To the greatest extent appropriate and feasible, SAMHSA will encourage the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.
SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: services and demonstrations. The evaluation required for a particular grant program depends on the type and purpose of the program. SAMHSA evaluates its service programs to provide information to program managers about the accountability of Federal funds. Its evaluations of demonstration programs generate new knowledge to lead the field in developing policies that improve services. Evaluation results from the two types of programs are used to improve the performance of SAMHSA and the field of substance abuse and mental health services. Program and evaluation staff work together to identify the questions or goals each grant program should address and to propose appropriate evaluation strategies.
The two types of grant programs represent two facets of SAMHSA's mission: service delivery and knowledge development. SAMHSA's leadership in the field depends on the successful interaction of these two facets of its mission. SAMHSA uses evaluation to identify effective approaches to prevention and treatment and uses service delivery funds to provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are the development, identification, and dissemination of effective strategies and systems for treatment and prevention.
SAMHSA is now implementing a new, integrated model of evaluation and planning. Strategic planning will identify priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is attempting to improve performance by identifying performance goals and indicators as part of its strategic planning process. The formulation of programmatic and evaluation priorities will include consultation with the SAMHSA and center advisory councils and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of evaluable programmatic objectives. Evaluations will show how well the overall grant programs have achieved their objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA's priorities and objectives.
This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way SAMHSA can best serve its customers by enhancing the quality of public substance abuse and mental health services.
In compliance with the Public Health Service (PHS) guidelines for the technical review of evaluations, SAMHSA has a standing review committee of PHS staff who are evaluation specialists from Agency for Health Care Policy and Research, Centers for Disease Control and Prevention, Health Resources and Services Administration, National Institutes of Health, and SAMHSA. Representatives of the Office of the Assistant Secretary for Health and the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members, and the SAMHSA evaluation coordinator serves as committee chair.
Evaluation project proposals are generally prepared by SAMHSA program staff in the various centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives; appropriateness and feasibility of the specifications for evaluation design and methods; appropriateness of the plans for dissemination of results; and use of previous relevant evaluations and existing program data systems.
Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policy priorities and evaluation program. The committee generally reviews proposals for broad, comprehensive evaluations of SAMHSA program activities, such as the cross-site evaluations of demonstration grant programs. It does not review the evaluation proposals of individual grantees.
Summary of FY 1994 Evaluations During FY 1994, SAMHSA completed six evaluations. The six studies reflect some of the population groups SAMHSA has identified as being in greatest need of substance abuse and mental health services. These include pregnant and postpartum women and their infants, children with serious emotional disturbance, high-risk youth, and the homeless mentally ill.
The evaluation of a demonstration program serving pregnant and postpartum women and their infants examined the results of improved coordination, availability, and accessibility of health and substance abuse-related services. A substantial number of the women served by these programs reduced their substance use, and their babies were generally healthy. SAMHSA will use these findings to encourage improvements in the performance of service programs for substance-abusing pregnant and postpartum women and their infants.
SAMHSA completed the design for an evaluation of a comprehensive community services program for children with serious emotional disturbance. The purpose of the evaluation is to collect information on client and systems outcomes. The findings should help both policymakers and service providers improve the performance and cost-effectiveness of mental health services available to children.
Another evaluation conducted this year examined the cost-effectiveness of a drug treatment enrichment program in curbing drug use among students receiving training at Job Corps Centers. Job Corps Centers are residential employment and training programs for high-risk youth. Students participating in the enrichment program showed a significant decrease in drug use compared with a control group. The findings of this study are useful in verifying the effectiveness of substance abuse treatment and in informing the field about appropriate substance abuse services for high-risk youth.
An evaluation of a demonstration program for homeless mentally ill adults is highlighted in chapter II of this report. This study examined the results of improved linkage, integration, and availability of housing and social and medical services. The program was successful in reducing homelessness and improving the mental health of study participants. These results are encouraging communities to better integrate services for the homeless mentally ill.
An evaluation of demonstration projects for community partnerships to prevent substance abuse has produced information on the successful implementation of substance abuse prevention approaches that employ a comprehensive, coordinated, communitywide strategy. Evaluation results of the impact of the partnership program are forthcoming. The Second Report to Congress on Alcohol and Drug Abuse Prevention presents preliminary results of the National Structured Evaluation (NSE). The NSE is a systematic assessment of substance abuse prevention approaches implemented across the country. Final results of this study, to be presented in a third and final report to Congress, will have important implications for improving the performance of substance abuse prevention programs.
Evaluations in Progress SAMHSA currently has 10 major evaluations under way in the following general areas: program accountability, evaluation of demonstrations, reinforcing behavioral health, and commitment to customer service. Each type is described below, with some examples.
Program accountability evaluations are undertaken to inform program management and help managers refine program operations. This is the primary type of evaluation conducted on SAMHSA's service grant programs. For example, the children's mental health service program has an ongoing evaluation. The evaluation will yield continuous information on program implementation and on outcomes for children and families served. This information will be used for reports to Congress, feedback to grantees, program development, and performance improvement.
Evaluations of demonstrations are designed to generate new knowledge for policy development. The primary purpose of SAMHSA's demonstration programs is to generate new knowledge to lead the field in the development of policies that improve services. For example, evaluation results on substance abuse prevention for high-risk youth will allow policymakers to draw inferences about the effectiveness of certain interventions for this population. The Job Corps evaluation will continue to assess an enriched substance abuse treatment program for adolescents. An evaluation of the program for access to community care and effective services and supports (ACCESS) will provide information for the design of ongoing service programs at the Federal, State, and local levels.
Other SAMHSA evaluations strive to reinforce behavioral health as a cornerstone of public health. In the context of managed care and health care reform, SAMHSA's interagency collaborations promote the integration of services and reinforce the role of alcohol, drug, and mental health services as a critical component of general health care. Illustrations of such activities include collaboration with the Administration on Children and Families to evaluate a substance abuse prevention program for pregnant and postpartum women and infants.
The National Treatment Improvement Evaluation Study (NTIES) was designed as one of a family of studies of substance abuse treatment, including two studies conducted by the National Institutes of Health (NIH). This collaboration will enhance SAMHSA's ability to draw inferences about treatment effectiveness. SAMHSA will continue working with the Department of Housing and Urban Development to design a method by which therapeutic communities can be incorporated into public housing areas. An HIV/AIDS mental health services demonstration program is jointly funded by NIH and HRSA.
Because of its commitment to the recipients of alcohol, drug abuse, and mental health services, SAMHSA uses evaluation to guide the field in improving the integration of services and their responsiveness to the needs of clients. For example, the evaluation of the community partnership program will determine the part-nership's ability to achieve integration and responsiveness at the user level. Primary purposes of the community coalitions demonstration program are to increase the efficiency of services at the community level and to enhance the service delivery system's responsiveness to its users. The evaluation of the program will assess the effectiveness of this approach.
SAMHSA has promulgated a comprehensive treatment model that includes a broad array of treatment and ancillary services targeted for individuals whose substance abuse has severely affected many domains of their lives. The services include housing, child care, transportation, social services assistance, legal aid, and mental and physical health screenings and treatment, as appropriate. Every SAMHSA grant announcement for substance abuse treatment will require that grantees implement the comprehensive treatment model. Evaluations will assess the effectiveness of the model.
New Directions for Evaluation SAMHSA is designing its evaluation activities to guide programmatic and policy decisions. The evaluation activities will complement the SAMHSA and PHS strategic plans and will respond to emerging trends such as managed care and health care reform. For example, in FY 1995 SAMHSA will initiate a program of State contracts to evaluate managed care and treatment outcome. The program will provide support to States to conduct short- and long-term studies at the State and provider levels of the effects of managed care on substance abuse treatment access, cost, and outcomes. SAMHSA will support States' use of evaluation methodologies including State financial, provider, and client databases.
These evaluation activities will reinforce the critical role of behavioral health in general health care. Results of the evaluations will ensure accountability and generate new knowledge. SAMHSA will use these results to improve integration of services and responsiveness to the needs of individuals suffering from or at risk for alcohol, drug abuse, and mental health problems.