Substance Abuse and Mental
Health Services Administration

Mission

To improve the quality and availability of prevention, treatment, and rehabilitation services in order to reduce illness, death, disability, and cost to society resulting from substance abuse and mental illnesses.

Evaluation Program

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to evaluating its overall programs and individual grant projects to assess the effectiveness of prevention, treatment, and rehabilitation approaches and systems of care; the accountability of federal funds; and the achievement of SAMHSA’s programmatic and policy objectives.

To the greatest extent possible, SAMHSA encourages the use of comparable data elements and instruments across its evaluations in order to work toward a comprehensive evaluation system and to minimize respondent burden.

SAMHSA conducts grant programs under a variety of legislative authorities. These authorities have generally been grouped into two types: (1) services and (2) knowledge development activity (KDA). The evaluation required for a particular grant program is dependent on the type and purpose of the program. SAMHSA evaluates each of its service programs so as to provide information to program managers about the accountability of federal funds.

SAMHSA has an integrated model of evaluation and planning. Strategic planning identifies priorities, such as managed care, that drive the development of grant programs and evaluations. In compliance with the Government Performance and Results Act (GPRA), SAMHSA is improving performance management by identifying annual performance objectives and goals. The formulation of programmatic and evaluation priorities includes consultation with SAMHSA Center Advisory Councils, with other HHS agencies, and with other experts in the fields of evaluation and service delivery. Early and continuous coordination of program planning and evaluation design will result in the articulation of program objectives that may be evaluated. Evaluations will demonstrate the extent to which the grant programs have achieved their overall objectives, and SAMHSA will translate these results into information that can be used for program and policy development. The strategic planning and policy development processes will then use these results to refine SAMHSA’s priorities and performance objectives.

Completed Evaluations

Evaluation of the Community Action Grant Program Phase I - Round I Grantees

The Community Action Grant Program (CAG) facilitates the adoption and implementation of exemplary practices in mental health services for adults with serious mental illness and children with severe emotional disturbance.  The grant program has two phases, each one year long.  In Phase I, applicants are funded to convene stakeholders and build consensus around the adoption of a chosen practice, as well as identify a funding source to fund the practice and implementation.  An evaluation of CAG grantees from the first year of the program found that all 17 grantees in the study reported achieving consensus around an exemplary mental health service practice.  Grantees engaged a wide array of stakeholders and employed various tactics and consensus building models to overcome barriers in moving towards consensus.  This preliminary evaluation serves as a stepping stone for other HIRI evaluation activities examining later rounds of the CAG program as well as an evaluation led by the Association for the Study and Development of Community, looking at trends in Phase I grantees since the program’s inception.  Information from all three evaluations will be used by CMHS staff to guide the development of the CAG program through activities such as revising the program announcement and reporting requirements, as well as improving technical assistance.
PIC ID: 7716; CONTACT: David Morrissett, 301-443-3653; PERFORMER: Human Interaction Research Institute, Los Angeles, CA

The Impact of Prior Physical and Sexual Victimization on Substance Abuse Treatment Outcomes

This technical report presents a secondary analysis of the National Treatment Improvement Evaluation Study (NTIES) data to assess the effects of pre-treatment physical and sexual violence on substance abuse treatment outcomes.  The relationship between treatment outcomes and past physical and sexual abuse are analyzed for males and females in five treatment modalities: methadone, non-methadone outpatient treatment, short-term residential treatment, long-term residential treatment, and treatment facilities located in correctional institutions.  Results indicate that victimization by physical and/or sexual violence was a frequent event in the life histories of persons entering substance abuse treatment that was part of NTIES.  Lifetime physical/sexual abuse was reported by three-fourths (73%) of the respondents in the outcome sample.  Of the 60 estimated mixed-model regression equations, the impact of prior victimization on treatment outcomes was significant in the expected direction, namely, victimization was associated with poorer outcomes at follow-up.
PIC ID: 5994.11; CONTACT: Ron Smith, 301-443-7730; PERFORMER: Caliber Associates, Fairfax, VA

In-Progress Evaluations

An Evaluation of PASRR and Mental Health Services for Persons in Nursing Facilities

The process of screening and determining whether nursing facility services and specialized mental health care are needed by nursing facility applicants and residents is called the Preadmission Screening and Resident Review (PASRR) program.  The PASRR program is a required component of each state’s Medicaid plan.  More than a decade after Congress enacted the PASRR program to prevent the inappropriate admission and retention of people with mental disabilities in nursing facilities, many concerns regarding the efficacy and effectiveness of the legislation remain.  Some national organizations say PASRR creates logistical barriers for persons requiring nursing facility placements and does not ensure access to appropriate (i.e., medically necessary) psychiatric services. The purposes of this study are to examine the: (1) intent, scope, and jurisdiction of PASRR; (2) implementation of PASRR at the state and nursing facility level; and (3) effect that PASRR has had on the identification of people with serious mental illness in nursing facilities.
PIC ID: 7729; EXPECTED COMPLETION: FY 2003; CONTACT: Judith Teich, 301-443-0588; PERFORMER: Lewin Group, San Francisco, CA

Consumer-Operated Services Program

The Consumer-Operated Services Program (COSP) is a four-year multisite evaluation of the effectiveness of consumer-run mental health services when provided as an adjunct to traditional mental health services. Begun in September 1998, seven projects are evaluating eight service programs in eight states, and a Coordinating Center is coordinating the multisite effort.  Each participant enrolled in COSP is randomized to one of two conditions: (1) consumer-operated services plus traditional mental health services, or (2) traditional mental health services alone.  Baseline enrollment ended on September 30, 2001 with over 1,900 participants enrolled.  This project will assess the degree to which consumer-operated services and traditional mental health services work together through application of a systems integration measure to data collected from the sites during site visits to be made by the Coordinating Center.
PIC ID: 7724; EXPECTED COMPLETION: FY 2004; CONTACT: Betsy McDonel-Herr, 301-443-3653; PERFORMER: Missouri Institute of Mental Health, St. Louis, MO

Cooperative Agreements to Evaluate Housing Approaches for Persons with Serious Mental Illness-Phase II

The Housing Initiative, a multi-site program initially funded in 1997, was designed to evaluate the effectiveness of different housing approaches for persons with serious mental illness.   The program was designed to be implemented in two phases.  This project, Phase II, is designed to complete the analysis of the cross-site study and disseminate the findings through a combination of reports, conference presentations, and publications.  The lack of available housing slots to place clients made it impossible for the individual sites to accrue their target samples within the time originally allocated.  Consequently, the six sites were all given one-year extensions to complete their data collection activities and the analysis of the cross-site evaluation.
PIC ID: 7726; EXPECTED COMPLETION: FY 2003; CONTACT: Pam Fischer, 301-443-3706; PERFORMER: Vanderbilt University, Nashville, TN

Evaluation of the Community Action Grant Program--Phase I, Rounds 2 and 3, and Phase II, Round 1

The Community Action Grant Program (CAG) facilitates the adoption and implementation of exemplary practices in mental health services for adults with serious mental illness and children with severe emotional disturbance.  The grant program has two phases, each one year long.  In Phase I, applicants are funded to convene stakeholders (e.g., consumers, policy-makers, providers) and build consensus around the adoption of a chosen practice, as well as identify a funding source to fund the practice implementation.  In Phase II, applicants who have successfully completed Phase I are given funding to maintain stakeholder support and aid in the implementation of the chosen practice (funds are not used for direct service delivery). This study will also evaluate consensus building processes, grantee reporting, and barriers to consensus building around the applicant’s chosen practice. The evaluation of Phase II grantees from Round 1 examines variables such as practice implementation, barriers to implementation, and commitments for the maintenance of exemplary practices.
PIC ID: 7716.1; EXPECTED COMPLETION: FY 2002; CONTACT: David Morrissett, 301-443-3653; PERFORMER: Human Interaction Research Institute, Los Angeles CA

Evaluation of the Comprehensive Community Mental Health Services for Children and their Families Program - Phase II

The Center for Mental Health Services (CMHS) is conducting an evaluation of 23 sites of the Comprehensive Community Mental Health Services for Children and their Families Program funded in fiscal years 1997 and 1998.  Evaluators hired at each site currently gather descriptive data on such measures as demographics, functional status, diagnosis, and risk factors of enrolled children and their families.  They are collecting outcome data in areas such as social functioning, emotional and behavioral strengths, substance abuse, school performance, and juvenile justice involvement in a representative sample of about 280 children per site.   To obtain the outcome data, children are assessed at the time of entry (i.e., baseline) and every six months for up to 36 months.  External evaluators visit the sites annually to study the development of systems of care according to system-of-care principles.  In addition, two systems of care are being compared with two usual service delivery systems to study the effectiveness of systems of care.
PIC ID: 7269.1; EXPECTED COMPLETION: FY 2003; CONTACT: Rolando L. Santiago, 301-443-1333; PERFORMER: ORC Macro, Calverton, MD

Evaluation of the Comprehensive Community Mental Health Services for Children and their Families Program - Phase III

CMHS began a third phase of the evaluation with a six-year evaluation of 22 new grant communities funded in fiscal years 1999 and 2000.  As of August 31, 2001, 2,333 children had enrolled in systems of care funded during this phase of the program.  Evaluators hired at each site are gathering descriptive data on measures such as demographics, functional status in the home, school and community, diagnostic status, and participation in the Child Health Insurance Program of enrolled children and their families.  Preliminary results indicate that with respect to diagnostic status, 31.8 percent of children are diagnosed with attention deficit hyperactivity disorders, 29.4 percent with mood disorders, 12.4 percent with oppositional defiant disorder, 4.8 percent with post-traumatic and acute stress disorder, 40 percent with conduct disorder, 3.8 percent with impulse control problems, and 14.0 percent with other disorders such as autistic and related, adjustment, psychosis, anxiety, among others.
PIC ID: 7269.2; EXPECTED COMPLETION: FY 2005; CONTACT: Rolando L. Santiago, 301-443-1333; PERFORMER: Macro International, Inc., Calverton, MD

Evaluation of the Effectiveness of Time-Limited Interventions for Homeless Families

The Center for Mental Health Services and the Center for Substance Abuse Treatment are collaborating in the funding and administration of a two-phase, five-year knowledge development initiative to document and evaluate the effectiveness of time-limited, intensive intervention strategies for providing mental health and/or substance abuse treatment, trauma recovery, housing, support, family preservation, and other wrap-around services to homeless mothers with psychiatric and/or substance use disorders who are caring for their dependent children.  In the Fall of 1999, Phase I began a two-year process with 14 study sites to develop population descriptions, document their intervention models, collaborate in the development of a cross-site interview instrument, and formulate cross- and individual-site evaluation plan designs.  In Phase II, which began in October 2001, study sites will conduct cross- and individual-site evaluations of their homeless families interventions, participate in developing the plan for analyzing the cross-site data, and begin collaboration on knowledge application and dissemination activities.
PIC ID: 7725; EXPECTED COMPLETION: FY 2004; CONTACT: Larry Ricards, 301-443-3706; PERFORMER: Policy Research Associates, Delmar, NY

Evaluation of the Effects of the California Mental Health Parity Act

In the past several years, federal and state legislation has been enacted to require parity benefits (equivalence between mental health benefits and general health care benefits in health insurance plans).  These legislative efforts to address parity are often limited in terms of their scope or application.  California enacted parity legislation in 1999, which became effective in 2000.  It provides for equal coverage for severe mental illnesses, and explicitly covers children with one or more mental disorders.  Further, unlike many other States, small businesses are not exempt from the law’s requirement.   Due to the size and complexity of California’s economy and health care market, it is important for SAMHSA to evaluate the effects of this parity mandate.  Consequently, the Center for Mental Health Services (CMHS) recently began a two-year study which will address such questions as:  What were specific issues/problems in the implementation of the parity legislation?  How have costs and utilization changed as a result of parity?  What are consumer, employer, insurer, and provider opinions about the effects of the law?
PIC ID: 7727; EXPECTED COMPLETION: FY 2003; CONTACT: Judith Teich, 301-443-0588; PERFORMER: Mathematica Policy Research, Inc., Plainsboro, NJ

Evaluation of the Effects of the Vermont Parity Act

In 1997, Vermont enacted parity legislation.  Effective January 1, 1998, or upon renewal, the law requires health plans to provide coverage for the treatment of “mental health conditions” and prohibits rates, terms, or conditions that impose a greater financial burden for access to treatment for mental illness than for physical illness.  Unlike many other parity laws, Vermont’s statute includes substance abuse and does not restrict parity to specific diseases.  Vermont’s parity law is more comprehensive than most others.   It, therefore, offers a unique opportunity to learn about the actual effects of a full parity mandate.  Some of the questions to be addressed in this evaluation include:  What mandates governed MH/SA benefits prior to parity?  What specifically does the law require?  Were there identifiable public/private cost shifts?  Has improved commercial reimbursement provided increased flexibility for use of funds from non-commercial sources (e.g., have dollars been redirected to prevention programming)?  Do consumers report improved access and fewer barriers to MH/SA services as a result of the law?
PIC ID: 7728; EXPECTED COMPLETION: FY 2002; CONTACT: Judith Teich, 301-443-0588; PERFORMER: Mathematica Policy Research, Inc., Plainsboro, NJ

Evaluation of the HHS Access to Community Care and Effective Services and Supports (ACCESS) Program

The ACCESS Program is a five year demonstration program that provides grants to nine States to develop integrated systems of treatment and supportive services and housing for homeless persons with serious mental illnesses. The goal of the program is to identify strategies for developing integrated service systems and to evaluate their effectiveness in providing services to homeless persons with serious mental illnesses. The evaluation of the ACCESS Program has two major components: The system-level evaluation identifies the different systems integration approaches, documents how they are implemented, and measures their impact on system capacity and system barriers. The client-level evaluation determines whether systems integration efforts result in improved service delivery, improvements in mental health and health status, rehabilitation, quality of life and permanent exit from homelessness.
PIC ID: 4980.1; EXPECTED COMPLETION: FY 2003; CONTACT: Randolph Frances, 301-443-3706; PERFORMER: R.O.W. Sciences, Inc., Rockville, MD

National Evaluation of the Comprehensive Community Mental Health Services for Children and their Families Program - Phase IV

The Center for Mental Health Services (CMHS) will start a new phase of the evaluation of the Comprehensive Community Mental Health Services for Children and their Families Program with a six-year evaluation of about 20 new grant communities funded in fiscal year 2002.    Evaluators hired at each site will gather descriptive data on measures such as demographics, functional status in the home, school, and community, diagnostic status, and resiliency and protective factors.  Local evaluations will collect outcome data in areas such as behavioral and emotional strengths, school performance, and juvenile justice involvement in a representative sample of children.  In addition, the national evaluation contractor will provide assistance in the development of a community-based management information system for monitoring service use and costs across child-serving sectors such as mental health, juvenile justice, child welfare, and education, especially those services and costs associated with an individualized care plan.
PIC ID: 7269.3; EXPECTED COMPLETION: FY 2008; CONTACT: Rolando L. Santiago, 301-443-1333; PERFORMER: ORC Macro, Calverton, MD

National Evaluation of the Safe Schools/Healthy Students

The Safe Schools/Healthy Students Interdepartmental Initiative (SS/HS) is a landmark effort supported by an unprecedented collaboration among the Departments of Education, Health and Human Services, and Justice.  The goals are to: (1) help students develop the skills and emotional resilience necessary to promote positive mental health, engage in prosocial behavior, and prevent violent behavior and drug use; (2) ensure that all students learn in a safe, disciplined, and drug-free environment; and (3) develop an infrastructure that will institutionalize and sustain integrated services.  This national evaluation of SS/HS is designed to document how coalitions and collaborations at the community level develop systemic change that results in beneficial outcomes for children, families, and schools.
PIC ID: 7717; EXPECTED COMPLETION: FY 2004; CONTACT: Office of Juvenile Justice, 202-514-4817; PERFORMER: Department of Justice, Juvenile Justice, Washington, DC

The Women, Co-Occurring Disorders, and Violence Study

SAMHSA is sponsoring a multi-site demonstration to assess the extent to which comprehensive, integrated, trauma-informed services can make a difference in the lives of women who have co-occurring mental health and substance abuse disorders and who have also experienced sexual and/or physical abuse.  The evaluation involves a quasi-experimental design comparing study participants to comparison groups using a common interview protocol at baseline, six months, and twelve months to assess variations in outcomes across the involved nine sites.  Study participants are currently being recruited and, thus, no outcome findings are available.
PIC ID: 7721; EXPECTED COMPLETION: FY 2003; CONTACT: Community Support Programs, 301-443-3653; PERFORMER: Policy Research Associates, Delmar, NY

Cross-site Evaluation of the Community Prevention Coalitions Demonstration Grant Program

This project will conduct a cross-site evaluation of the Community Prevention Coalition (CPC) grant program over a five-year period. The goals of the evaluation are to assess: (1) the extent to which coalitions of service organizations created by the program are able to implement comprehensive and coordinated alcohol, tobacco, and other drug (ATOD) prevention programs through an expanded and non-duplicative service delivery base; (2) the extent to which coalitions are able to achieve measurable reductions in the incidence and prevalence of ATOD use, and improvements in ATOD-related social, health and safety consequences among all age, ethnic and vulnerable groups within grantee communities; (3) the contributions of various prevention strategies to site-specific outcomes and to general prevention outcomes in those sites where measurable changes are made; and (4) those characteristics common to coalitions where measurable changes are achieved, as well as those that differentiate successful coalitions from unsuccessful ones. The contractor will be required to collect process and outcome data for analysis from coalitions. In addition, data collected locally by grantees and archived community-wide indicators will be used.
PIC ID: 6042; EXPECTED COMPLETION: FY 2002; CONTACT: Shakeh Kaftarian, 301-443-9136; PERFORMER: Cosmos Corporation, Bethesda, MD

Evaluation of Opioid Treatment Program Accreditation Project

The Center for Substance Abuse Treatment (CSAT) and other federal agencies are developing a regulatory framework for opioid substitution therapy that will change the current regulatory system for monitoring opioid treatment programs (OTPs) to an accreditation/regulatory approach.  Over a five year period, CSAT will conduct and evaluate a phase-in program to obtain information that will aid in the full implementation of the new system. The goal of the evaluation is to obtain information that can guide the full national implementation of the new system for OTPs through systematic study of the processes, barriers, costs associated with a change from regulatory to an accreditation/regulatory process, and various other impacts.
PIC ID: 6739; EXPECTED COMPLETION: FY 2002; CONTACT: Herman Diesenhaus, 301-443-6575; PERFORMER: Research Triangle Institute, Research Triangle Park, NC

National Evaluation Data Services

The National Evaluation Data Services (NEDS) was established to increase the number of scientifically based analyses to answer vital questions in the substance abuse treatment field.   NEDS represents part of an overall CSAT evaluation strategy that builds upon prior findings and seeks to mine existing data whose potential has not been fully explored.  One objective of the NEDS project is to provide a wide array of data management and scientific support services across various program and evaluation activities, including an information system of available data related to the evaluation of substance abuse treatment. Another objective is to provide CSAT with a flexible analytical capability to use existing data to address policy-relevant questions about substance abuse treatment, and to clearly articulate the resulting analyses as they relate to policy through fact sheets, evaluation summaries, technical reports, methods development documents and technical guidance documents.
PIC ID: 5994.4; EXPECTED COMPLETION: FY 2002; CONTACT: Ron Smith, 301-443-7730; PERFORMER: Caliber Associates, Fairfax, VA

Persistent Effects of Treatment Studies (PETS)

The purpose of PETS is to evaluate the long-term effectiveness (up to thirty-six months) of substance abuse treatment services provided through a series of CSAT grants and cooperative agreements, and to conduct a number of special studies and policy analyses that address specific drugs of abuse, methods of treatment, populations, or policy issues.  The primary follow-up studies will involve obtaining information at selected follow-up periods for persons who have completed the index treatment episode and for whom historical, intake and treatment exit data are available.
PIC ID: 6738; EXPECTED COMPLETION: FY 2002; CONTACT: Ron Smith, 301-443-7730; PERFORMER: Westat, Inc., Rockville, MD

Primary Care Research in Substance Abuse and Mental Health Services for the Elderly (PRISMe)

SAMHSA has developed a multisite study to compare the effectiveness of service delivery models that treat older adults with mental health and substance abuse (MH/SA) problems in primary care as opposed to specialty MH/SA settings.  The study hopes to identify differences in clinical and cost outcomes between models referring consumers to specialty MH/SA services outside the primary care setting and those providing such services within the primary care settings.  This four-year study plans to randomize close to 3,000 patients with MH/SA problems to either integrated or referral models of MH/SA care.  This initiative includes the active collaboration of all three SAMHSA centers, with the Center for Mental Health Services (CMHS) as the lead, the Health Resource Services Administration’s Bureau of Primary Care, the Department of Veterans Affairs, and the Centers for Medicaid and Medicare Services.
PIC ID: 7722; EXPECTED COMPLETION: FY 2003; CONTACT: Betsy McDonel-Herr, 301-443-3653; PERFORMER: Harvard Medical School, Laboratory of Community Psychiatry, Boston, MA

State Substance Abuse Managed Care Evaluation Program

The State Substance Abuse Managed Care Evaluation Project will evaluate the effects of managed care on mental health and substance abuse services in five states. Each of these five state studies will inform future behavioral health care policy.
PIC ID: 6454; EXPECTED COMPLETION: FY 2002; CONTACT: Joan Dilonardo, 301-443-8555; PERFORMER: Brandeis University, Waltham, MA

Treatment Improvement Protocols Field Evaluation

More than 25 Treatment Improvement Protocols (TIPs) have been distributed to the public. The purpose of TIPs is to transfer protocols and guidelines for the treatment of alcohol and other drug (AOD) abuse from acknowledged clinical, evaluation and administrative experts to the nation’s AOD treatment field.  Institution of a field evaluation is needed to:  (1) assess the extent to which target audiences realize, read and implement TIPs; and (2) learn ways to strengthen the development, formatting, marketing, dissemination, use and evaluation of TIPs to ensure effective use of future federal resources devoted to TIPs. The overall objective of the proposed evaluation is to learn the impact of TIPs on the substance abuse treatment field. The study will examine methods of dissemination used by CSAT, the success of those methods in reaching the target audiences, users’ perceptions of the value of TIPs, decisions to implement TIPs, and the successes, correlates and barriers associated with implementation.
PIC ID: 6737; EXPECTED COMPLETION: FY 2002; CONTACT: Kevin Mulvey, 301-443-9472; PERFORMER: Johnson, Bassin and Shaw, Inc., Silver Spring, MD