|
Office of the Assistant Secretary for Planning and Evaluation |
| |
|
Policy Information Center |
MISSION: To improve the quality and availability of prevention, treatment, and rehabilitation services in order to reduce illness, death, disability, and cost to society resulting from substance abuse and mental illnesses.
Evaluation Program
The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its overall programs and individual grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care; the accountability of Federal funds; and the achievement of SAMHSA's programmatic and policy objectives.
To the greatest extent appropriate and feasible, SAMHSA encourages the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.
SAMHSA conducts grant programs under a variety of legislative authorities. These authorities can generally be grouped into two types: (1) services and (2) knowledge development activity (KDA). The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA evaluates each of its service programs so as to provide information to program managers about the accountability of Federal funds. In addition, the evaluations of KDA programs will generate new knowledge to lead the field in the development of policies that improve services.
The two types of grant programs (service delivery and KDA) represent the two facets of SAMHSA's mission. SAMHSA's leadership in the field depends on the successful interaction of these two facets. Through KDAs, SAMHSA must identify effective approaches to prevention, treatment, and rehabilitation. Through service delivery funds, SAMHSA must provide incentives to the field to implement effective approaches. Major emphases of SAMHSA's mission are to develop, identify, and disseminate effective strategies and systems for prevention, treatment, and rehabilitation.
SAMHSA is implementing an integrated model of evaluation and planning. Strategic planning identifies priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is improving performance management by identifying annual performance objectives and measures. The formulation of programmatic and evaluation priorities includes consultation with SAMHSA Center Advisory Councils, and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that may be evaluated. Evaluations will demonstrate the extent to which the grant programs have achieved their overall objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA's priorities and performance objectives.
This evaluation policy will help SAMHSA achieve its goal of continually informing policy and program development with knowledge culled from past performance. In this way, SAMHSA can best serve its customers by enhancing the quality of publicly-funded substance abuse and mental health services.
In compliance with the Public Health Service (PHS) guidelines for the technical review of evaluations, SAMHSA has established a standing committee of PHS staff who are evaluation specialists. Representatives of the Office of the Assistant Secretary for Planning and Evaluation serve as ex officio members of the committee. The SAMHSA evaluation officer is the committee chair. The committee does not generally review the evaluation proposals of individual grantees; rather, it reviews proposals for broader, more comprehensive evaluations, such as the cross-project evaluations of grant programs.
Evaluation project proposals are generally prepared by SAMHSA program staff in the various Centers. The standing committee reviews each proposal on the following criteria: clarity of evaluation objectives and research questions, appropriateness and feasibility of the specifications for evaluation design and methods, appropriateness of the plans for dissemination of results, and use of previous relevant evaluations and existing program data systems. Each proposal must clearly state the relationship of the evaluation to SAMHSA's overall policies, priorities, and evaluation program.
Fiscal Year 2000 Evaluation Reports
The Comprehensive Community Mental Health Services for Children and their Families Program provides grants to improve and expand local systems of care designed to meet the individualized needs of children and adolescents with a serious emotional disturbance and their families. This report presents data from the third year of the evaluation of the children's services program, based on information collected through August 1998 from 22 grantees that were funded for their first of five years of support. The evaluation focuses on programs that established systems of care for approximately 34,377 children and their families. Presented in the report are two types of evaluation data, (1) outcome data based on project site evaluations of a selected group of 200 to 400 children assessed at intake, 6 months, 1 year, and annually thereafter for as long as they remained in the program. Outcome measures examined included an assessment of the child's clinical and social functioning, educational performance, and stability of living arrangements, coupled with an examination of family and child ratings of the services provided; and (2) descriptive data (e.g., demographic information, diagnostic status, functional characteristics, and referral sources) obtained at the time that children entered services.
AGENCY SPONSOR: Center for Mental Health Services
FEDERAL CONTACT: Rolando L. Santiago, 301-443-3808
PIC ID: 7269
PERFORMER: Macro International, Inc., Calverton, MD
This project will conduct a cross-site evaluation of the Community Prevention Coalition (CPC) grant program over a five-year period. The goals of the evaluation are to assess: (1) the extent to which coalitions of service organizations created by the program are able to implement comprehensive and coordinated alcohol, tobacco, and other drug (ATOD) prevention programs through an expanded and non- duplicative service delivery base; (2) the extent to which coalitions are able to achieve measurable reductions in the incidence and prevalence of ATOD use, and improvements in ATOD-related social, health and safety consequences among all age, ethnic and vulnerable groups within grantee communities; (3) the contributions of various prevention strategies to site-specific outcomes and to general prevention outcomes in those sites where measurable changes are made; and (4) those characteristics common to coalitions where measurable changes are achieved, as well as those that differentiate successful coalitions from unsuccessful ones. The contractor will be required to collect process and outcome data for analysis from coalitions. In addition, data collected locally by grantees and archived community-wide indicators will be used.
EXPECTED DATE OF COMPLETION:10/31/2001
AGENCY SPONSOR: Center for Substance Abuse Prevention
FEDERAL CONTACT: Shakeh Kaftarian, 301-443-9136
PIC ID: 6042
PERFORMER: Cosmos Corporation, Bethesda, MD
The Center for Substance Abuse Treatment (CSAT) and other federal agencies are developing a regulatory framework for opioid substitution therapy that will change the current regulatory system for monitoring opioid treatment programs (OTPs) to an accreditation/regulatory approach. Over a five year period, CSAT will conduct and evaluate a phase-in program to obtain information that will aid in the full implementation of the new system. The goal of the evaluation is to obtain information that can guide the full national implementation of the new system for OTPs through systematic study of the processes, barriers, costs associated with a change from regulatory to an accreditation/regulatory process, and various other impacts.
EXPECTED DATE OF COMPLETION:09/29/2002
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Herman Diesenhaus, 301-443-6575
PIC ID: 6739
PERFORMER: Research Triangle Institute, Research Triangle Park, NC
The Center for Mental Health Services (CMHS) is conducting an evaluation of 23 sites of the Comprehensive Community Mental Health Services for Children and their Families Program funded in fiscal years 1997 and 1998. Evaluators hired at each site currently gather descriptive data on such measures as demographics, functional status, diagnosis, and risk factors of enrolled children and their families as well as outcome data in such areas as social functioning, substance abuse, school performance, and juvenile justice involvement in a representative sample of 225 children. The children are assessed every 6 months for up to 36 months. A service-level substudy includes a provider survey and measures of multisector service contacts, client satisfaction, and adherence to a service plan. External evaluators visit the sites annually to assess implementation of services according to system care goals and philosophy. Three systems of care are being compared with three usual service systems to study how systems of care develop over time. Preliminary analyses indicate that 21 percent of referrals come from juvenile justice agencies, 21 percent from mental health providers, 20 percent from parents, 19 percent from schools, 9 percent from child welfare programs, and the remaining 10 percent from self or other sources.
EXPECTED DATE OF COMPLETION:09/29/2001
AGENCY SPONSOR: Center for Mental Health Services
FEDERAL CONTACT: Rolando L. Santiago, 301-443-3808
PIC ID: 7269.1
PERFORMER: Macro International, Inc., Calverton, MD
The Center for Mental Health Services (CMHS) will begin an evaluation of 20 new sites of the Comprehensive Community Mental Health Services for Children and their Families Program funded on September 30, 1999. Evaluators hired at each site will gather descriptive data on such measures as demographics, functional status, and participation in the Child Health Insurance Program of enrolled children and their families. These evaluators also will collect outcome data in such areas as social functioning, substance abuse, school performance, and juvenile justice involvement in a representative sample of children. External evaluators will visit the sites every 18 months to assess service implementation according to system of care program goals and philosophy. Local sites will receive technical assistance on how to create logic models to describe and self-monitor the development of their systems of care. The evaluation contractor will develop an Internet-based system to enter and monitor data collected at local sites to disseminate findings through Web-based technology. The evaluation contractor will also conduct a study on the effectiveness of specific services, treatments, or supports delivered within a selected system of care site.
EXPECTED DATE OF COMPLETION:09/29/2003
AGENCY SPONSOR: Center for Mental Health Services
FEDERAL CONTACT: Rolando L. Santiago, 301-443-3808
PIC ID: 7269.2
PERFORMER: Macro International, Inc., Calverton, MD
The ACCESS Program is a five year demonstration program that provides grants to nine States to develop integrated systems of treatment and supportive services and housing for homeless persons with serious mental illnesses. The goal of the program is to identify strategies for developing integrated service systems and to evaluate their effectiveness in providing services to homeless persons with serious mental illnesses. The evaluation of the ACCESS Program has two major components: The system-level evaluation identifies the different systems integration approaches, documents how they are implemented, and measures their impact on system capacity and system barriers. The client-level evaluation determines whether systems integration efforts result in improved service delivery, improvements in mental health and health status, rehabilitation, quality of life and permanent exit from homelessness.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Center for Mental Health Services
FEDERAL CONTACT: Randolph Frances, 301-443-3706
PIC ID: 4980.1
PERFORMER: R.O.W. Sciences, Inc., Rockville, MD
The National Evaluation Data Services (NEDS) was established to increase the number of scientifically based analyses to answer vital questions in the substance abuse treatment field. NEDS represents part of an overall CSAT evaluation strategy that builds upon prior findings and seeks to mine existing data whose potential has not been fully explored. One objective of the NEDS project is to provide a wide array of data management and scientific support services across various program and evaluation activities, including an information system of available data related to the evaluation of substance abuse treatment. Another objective is to provide CSAT with a flexible analytical capability to use existing data to address policy- relevant questions about substance abuse treatment, and to clearly articulate the resulting analyses as they relate to policy through fact sheets, evaluation summaries, technical reports, methods development documents and technical guidance documents.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Ron Smith, 301-443-7730
PIC ID: 5994.4
PERFORMER: Caliber Associates, Fairfax, VA
The purpose of PETS is to evaluate the long-term effectiveness (up to thirty-six months) of substance abuse treatment services provided through a series of CSAT grants and cooperative agreements, and to conduct a number of special studies and policy analyses that address specific drugs of abuse, methods of treatment, populations, or policy issues. The primary follow-up studies will involve obtaining information at selected follow-up periods for persons who have completed the index treatment episode and for whom historical, intake and treatment exit data are available.
EXPECTED DATE OF COMPLETION:09/30/2002
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Ron Smith, 301-443-7730
PIC ID: 6738
PERFORMER: Westat, Inc.
Rockville, MD
The State Substance Abuse Managed Care Evaluation Project will evaluate the effects of managed care on mental health and substance abuse services in five States. Each of these five State studies will inform future behavioral health care policy.
EXPECTED DATE OF COMPLETION:09/30/2001
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Joan Dilonardo, 301-443-8555
PIC ID: 6454
PERFORMER: Brandeis University, Waltham, MA
More than 25 Treatment Improvement Protocols (TIPs) have been distributed to the public. The purpose of TIPs is to transfer protocols and guidelines for the treatment of alcohol and other drug (AOD) abuse from acknowledged clinical, evaluation and administrative experts to the nation's AOD treatment field. Institution of a field evaluation is needed to: (1) assess the extent to which target audiences realize, read and implement TIPs; and (2) learn ways to strengthen the development, formatting, marketing, dissemination, use and evaluation of TIPs to ensure effective use of future Federal resources devoted to TIPs. The overall objective of the proposed evaluation is to learn the impact of TIPs on the substance abuse treatment field. The study will examine methods of dissemination used by CSAT, the success of those methods in reaching the target audiences, users' perceptions of the value of TIPs, decisions to implement TIPs, and the successes, correlates and barriers associated with implementation.
EXPECTED DATE OF COMPLETION:12/31/2001
AGENCY SPONSOR: Center for Substance Abuse Treatment
FEDERAL CONTACT: Kevin Mulvey, 301-443-9472
PIC ID: 6737
PERFORMER: Johnson, Bassin and Shaw, Inc., Silver Spring, MD