Case Study: Early Assessment of the Mental Health Block Grant Set-Aside Program for Addressing First Episode Psychosis and Other Early Serious Mental Illness

Marcela Horvitz-Lennon, Joshua Breslau, Deborah Scharf, Madeline Doyle, Nupur Nanda, Daniela Kusuke, Justin W. Timbie, and Virginia Kotzias

RAND Health

---

ABSTRACT

ASPE is releasing a report on the initial evaluation of the implementation of the Mental Health Block Grant (MHBG) 5% Set Aside for early intervention programs for Early Serious Mental Illness (ESMI). Twelve states were selected and classified into three tiers according to their stage of implementation of ESMI programs and intended use of the set-aside funds, representing a gradient from most advanced in their implementation (Tier 1) to least (Tier 3), depending on whether they already had an early intervention program, had a prior RAISE study site, participated in the Medicaid expansion, or their degree of centralization in governance.

DISCLAIMER: The opinions and views expressed in this report are those of the authors. They do not necessarily reflect the views of the Department of Health and Human Services, the contractor or any other funding organization.

"

Acknowledgments

The authors are deeply grateful to Kirsten Beronio, Kristina West, Jessamy Taylor, and Monica Feit, of the Office of the Assistant Secretary for Planning and Evaluation (ASPE) at the U.S. Department of Health and Human Services for their support and guidance at all stages of this project. We would also like to thank Robert Heinssen and Lisa Colpe from the National Institute of Mental Health for their assistance in designing the project and interpreting its findings.

We also thank our two peer reviewers, whose thoughtful reviews and comments greatly improved this report.

We extend our sincerest gratitude to the representatives from the 22 programs who participated in research discussions during the course of the study. We are indebted to them for sharing their insights candidly and for the valuable input they provided. The findings reported here are enriched because of their participation.

This research was conducted under contract with ASPE. The project was conducted with ASPE input, however, the material contained in this report is the responsibility of the research team alone, and does not necessarily reflect the views of the sponsoring agency.

Acronyms

The following acronyms are mentioned in this report and/or appendix.

ACT	Assertive Community Treatment
ANSA	Adult Needs and Strengths Assessment
ASPE	HHS Office of the Assistant Secretary for Planning and Evaluation
AVATAR	EHR Software
BPRS	Brief Psychiatric Rating Scale
Cal Poly	California Polytechnic State University
CANS	Child and Adolescent Needs and Strengths Assessment
CBTp	Cognitive Behavioral Therapy for Psychosis
CI	Confidence Interval
CMBHS	Texas Clinical Management for Behavioral Health Services
CMHC	Community Mental Health Center
CSB	Virginia Community Service Board
CSC	Coordinated Specialty Care
DCF	Connecticut Department of Children and Families
DCHS	Virginia Department of Community and Human Services
DHCS	California Department of Health Care Services
DMHAS	Connecticut Department of Health Care Services
DUP	Duration of Untreated Psychosis
EASA	Early Assessment and Support Alliance
EHR	Electronic Health Record
EP-TAP	North Carolina Early Psychosis Technical Assistance Program
ePEP	Texas Enhanced Program for Early Psychosis
EPPIC	Early Psychosis Prevention and Intervention Center
ESMI	Early Serious Mental Illness
FEP	First Episode Psychosis
FOT	California First Onset Psychosis Team
FSP	Full Service Partnership
FY	Fiscal Year
GAP	Governor's Access Plan
GR	General Revenue
HHS	U.S. Department of Health and Human Services
IOL	Connecticut Institute of Living
JMHC	Wisconsin Journey Mental Health Center
LMHA	Texas Local Mental Health Authority
MASQ	Mood and Anxiety Symptoms Questionnaire
MCO	Managed Care Organization
MHBG	Mental Health Block Grant
MHMRA	Mental Health and Mental Retardation Authority
NAVIGATE	CSC treatment program for people experiencing FEP
NIMH	HHS National Institute of Mental Health
NOMS	National Outcomes Measurement System
NSDUH	National Survey on Drug Use and Health
NTBHA	North Texas Behavioral Health Authority
OASIS	North Carolina Outreach and Support Intervention Services
PANSS	Positive and Negative Symptom Scale
PHQ	Patient Health Questionnaire
PIER	Portland Identification and Early Referral
PREP	Prevention and Recovery in Early Psychosis
PRS	Psychiatric Rehabilitation Services
RAISE	Recovery After an Initial Schizophrenia Episode
RSN	Washington Regional Support Networks
SAMHSA	HHS Substance Abuse and Mental Health Services Administration
SFS	Social Functioning Scale
SMHA	State Mental Health Authority
SMI	Serious Mental Illness
STEP	Connecticut Specialized Treatment Early in Psychosis
TAY	Transitional Age Youth
TEDS	Treatment Episode Data Set
TESS	Transitional Engagement Supportive Services
TIP	Transitions to Independence Process
TOPPS	Treatment Outcome Program Performance System
TRAILS	Transitioning Adults into Living Successfully
UNC	University of North Carolina
UT	University of Texas
Wake STEP	North Carolina Wake Schizophrenia Treatment and Evaluation Program

Executive Summary

The 5-percent set-aside policy was conceived as a legislative initiative to promote greater access to evidence-based services for people with early serious mental illness (ESMI), a population with a large unmet need for health care that stems from its transitional age, complex health care needs, and inadequate insurance coverage. The legislation directed the U.S. Department of Health and Human Services (HHS) Substance Abuse and Mental Health Services Administration (SAMHSA) to require that states set-aside 5 percent of their Mental Health Block Grant (MHBG) allocation to develop or enhance existing evidence-based programs for this population starting in 2014. SAMHSA has collaborated closely with the HHS National Institute of Mental Health (NIMH) throughout the implementation of the policy.

Although states were encouraged to fund treatment programs that would deliver a specific multicomponent model--coordinated specialty care (CSC)--and serve young individuals with first episode (non-affective) psychosis (FEP), they were given the options of funding other evidence-based interventions and targeting their programs to individuals with ESMI other than FEP (i.e., early stages of affective psychotic disorders such as bipolar disorder, and early stages of any non-psychotic serious mental illness (SMI) with a gradual onset, such as obsessive-compulsive disorder). States were encouraged to leverage funds through inclusion of services reimbursed by Medicaid or private insurance, and they were informed that the policy included an expectation that program effectiveness needed to be demonstrated through a formal evaluation.

Given the latitude that states have with respect to using the set-aside funds, NIMH, SAMHSA, and the HHS Office of the Assistant Secretary for Planning and Evaluation (ASPE) sought to better understand how the funds are being used within states, particularly the strategy used by each of the states to implement and evaluate the policy. RAND conducted a series of case studies across 12 states selected by NIMH and ASPE to provide an early assessment of the set-aside policy. States were classified into three tiers according to their stage of implementation of ESMI programs and intended use of the set-aside funds, representing a gradient from most advanced in their implementation (Tier 1) to least (Tier 3). The states were California, Connecticut, and New York (Tier 1); Idaho, North Carolina, Texas, Virginia, and Wisconsin (Tier 2); and Colorado, Nevada, Iowa, and Washington (Tier 3).

The case study states varied with regard to a number of characteristics and developments that can affect the implementation of the policy or its success (e.g., whether states operated ESMI programs or had been study sites for an NIMH-funded initiative on the feasibility and effectiveness of CSC prior to the launch of the policy, states' Medicaid expansion and 1915(i) adoption status, the extent to which other funds were used to develop or expand services, degree of decentralization).

The case study states also varied in their strategies for implementing the set-aside policy. While only Tier 1 states with existing programs could use the funds to expand those programs, Tier 2 and Tier 3 states used the funds to develop new programs. Most states funded one or two grantees, with one team per grantee, but there were some notable exceptions (for example, Virginia funded eight grantees, with one team per grantee). There were differences among the case study states in the degree of prescriptiveness of the State Mental Health Authority (SMHA) with regard to the model to be implemented, target population, training curriculum, and uses of the set-aside funds. The SMHAs also varied in regard to having an explicit expectation that the set-aside funds would be supplemented by third-party reimbursement when services were rendered to insured clients. ESMI programs in most states served youth and young adults, with the majority of programs serving individuals within or near the age range of 15-25. Maximum or expected program caseloads varied across states, spanning from 15 individuals to 151 individuals per team. Most grantees leveraged existing partnerships and collaborations, both for referrals and the design/implementation of the evaluation component. In most states, the SMHA deferred to the grantees to design the evaluation component, and most grantees were still in the planning stages by the time these case studies were conducted. The evaluation plans were typically focused more on the measurement of process of care and outcomes than on structure, but all grantees were aware of the need to monitor fidelity.

Several barriers to implementation were identified in the course of these case studies, including the challenge of developing programs for a low-incidence disorder, dealing with eligibility criteria that are narrower than for most social service programs, hiring appropriately trained staff in the setting of the workforce shortages that are common in public mental health service systems across the county, and conducting program evaluations in the setting of limited data collection and analysis capability. Some facilitative factors were also identified, including strong state guidance; existing programs for transitional age youth and provider networks; and existing expertise in CSC and other ESMI models.

A number of themes emerged during these case studies that could inform evaluation approaches for CSC and other ESMI programs being implemented across the country. These pertain to maintaining model fidelity, selecting process and outcomes domains and measures, measurement of program quality and effectiveness, and the value of monitoring the referral process.

In conclusion, all case study states embraced the set-aside policy as a mechanism for developing or expanding services for people with FEP and other ESMI, but the case studies revealed wide variation among states in how funds were used. While most states implemented CSC programs or modified existing programs to become CSC-like, many expanded their clinical population focus to disorders beyond FEP, mainly to broaden access and enhance sustainability. States varied in the sophistication and state of implementation of their evaluation plans, but all of them were committed to the goal of evaluating program fidelity and effectiveness. Although most of the states had not yet developed sustainable models for their programs and were reliant on continuing MHBG support, all states expressed interest in eventually tapping into third-party reimbursement to cover at least some operational costs.

It is possible to conclude at this early stage that the set-aside policy is improving access to services for individuals experiencing their first episodes of SMI. To ensure success of the policy, however, there is a need for ongoing federal guidance on best practices for program implementation and evaluation.

Introduction and Background

Origins of the 5-Percent Set-Aside Policy to Address Needs of Persons with Early Serious Mental Illness

Scientific Motivation, Vision, and Goals for the Program

The 5-percent set-aside policy was conceived as a legislative initiative to promote greater access to evidence-based services for people with early serious mental illness (ESMI),^1,a population with a large unmet need for health care that stems from its transitional age (i.e., transitioning from child to adult treatment programs), complex health care needs, and inadequate insurance coverage.^{2, 3} The legislation directed U.S. Department of Health and Human Services (HHS) Substance Abuse and Mental Health Services Administration (SAMHSA) to require that states set-aside 5 percent of their Mental Health Block Grant (MHBG) allocation to support evidence-based programs for this population starting in fiscal year (FY) 2014. To address potential concerns that the policy would divert funds from existing programs, Congress provided an increase to the MHBG over the FY 2013 level to help states meet the new requirement. SAMHSA has collaborated closely with the HHS National Institute of Mental Health (NIMH) throughout the implementation of the policy.

The House and Senate reports that accompanied the legislation made generic reference to treatment programs for first episode psychosis (FEP). However, in its guidance to states,¹ SAMHSA made reference to a specific treatment model, coordinated specialty care (CSC) for FEP, directing states to a white paper released by NIMH in April 2014 titled Evidence-Based Treatments for First Episode Psychosis: Components of Coordinated Specialty Care.^{4, b} CSC is an early-intervention program that has been widely adopted by a number of industrialized countries with health care systems that, compared with the United States health care system, permit greater flexibility in the financing of complex interventions.^{5, 6} The intervention utilizes a multidisciplinary team-based and outreach-capable approach to recruit, engage, and treat young persons ages 15-25 with FEP who have been ill for no more than five years. CSC comprises several evidence-based treatment components, including assertive case management, individual or group cognitive-behavioral psychotherapy, supported employment and education services, family education and support, and low doses of selected antipsychotic agents. The approach is grounded in a patient-centered, youth-friendly, recovery-oriented paradigm that emphasizes shared decisionmaking and aims at preventing social and occupational disability.⁴ Although CSC shares some important features with assertive community treatment (ACT)--the evidence-based practice for the treatment of chronic serious mental illness (SMI)⁷ (team-based nature, use of outreach, and some amount of case management)--CSC has a greater focus on education and employment, and it is more time-limited and office-based than ACT.

The set-aside policy came about as a result of a number of recent developments in the scientific and public policy arenas.⁸

First, a critical mass of scientific evidence has emerged on the potential public health significance of shortening the duration of untreated psychosis (DUP) among people with FEP, given suggestive evidence of a positive association between interventions designed to reduce DUP and both short-term and long-term outcomes including symptom severity and disability.^{9, 10, 11, 12} Evidence from abroad and some academic centers in the United States suggests that comprehensive early intervention programs have beneficial short-term effects.^{6, 13, 14} At the same time, the NIMH-funded Recovery After an Initial Schizophrenia Episode (RAISE) initiative launched in 2008 (http://www.nimh.nih.gov/raise) has begun generating evidence on the feasibility and short-term effectiveness of CSC in community mental health centers (CMHCs) in the United States through its two research programs (Early Treatment Program/Navigate and Implementation Evaluation Study [NAVIGATE]/Connection).^{15, 16, 17}

Additionally, a series of high-profile acts of violence perpetrated by young persons with untreated SMI (SMI) has raised awareness of the size of the unmet need for high-quality care for this population.^{18, 19, 20}

The implementation of the ACA and expansion of parity laws are expected to have a significant impact on coverage and access to critical services for people with ESMI and may facilitate entry into programs tailored to meet their needs. Key provisions of these laws include the option of Medicaid expansion; the 2010 provision that allows young adults 19-25 to remain enrolled as dependents of their parents' insurance policies; elimination of exclusions for pre-existing conditions; access to subsidized private insurance; inclusion of mental health and substance abuse benefits in the package of essential health benefits; extension of parity protections to marketplace plans and Medicaid managed care; and enhancement of the optional Medicaid authority 1915(i) that allows states to provide home and community-based services.^{2, 21, 22} However, there is also recognition that an infusion of public funds is needed to furnish the public mental health system with a network of evidence-based programs for people with ESMI and ensure their sustainability. Moreover, although more ESMI services may now be paid for by Medicaid or private insurance, policymakers are aware that some key services for this population are unlikely to be well covered (or covered at all) by Medicaid or other insurance.²³

SAMHSA's Requirements for the Use of Set-Aside Funds

In its guidance to states,¹ SAMHSA required states to revise their two-year MHBG plan to describe the specific ESMI disorders they sought to address and how they would utilize the set-aside funding. States were encouraged to fund CSC programs for people with FEP but were given the options of funding other evidence-based interventions and targeting their programs to individuals with ESMI other than FEP. States could use the funding either to develop new programs or, for states with previous treatment infrastructure, enhance existing programs. States were also encouraged to leverage funds "through inclusion of services reimbursed by Medicaid or private insurance."

The guidance informed states that SAMHSA and NIMH would "hold a national webinar to inform states of the evidence-based components of CSC for FEP" and that SAMHSA would make technical assistance and resources available to states during the implementation period.c The guidance also informed states that the set-aside policy "includes an initiative for data collection related to demonstrating program effectiveness," with language indicating that "technical assistance and guidance on the expectations for data collection and reporting" would follow.

SAMHSA explicitly recognized that states would vary in their capacity to implement the new programming because of variation in the actual size of the 5-percent allocation. The agency also recognized that "states may need to dedicate the first year to planning, training, and/or infrastructure-development while targeting program implementation to the second year of the plan." States' MHBG plan revisions were required to provide information on the need for ESMI services and justification for the selection of the target population, the activities proposed, and the budget.

Purpose of the RAND Study

Given the latitude that states have with respect to use of the set-aside, NIMH, HHS SAMHSA, and the HHS Office of the Assistant Secretary for Planning and Evaluation (ASPE) sought to better understand how the funds are being used within states, particularly the extent to which the policy has led to the development or expansion of evidence-based ESMI care. RAND conducted a series of case studies across 12 states to provide an early assessment of the set-aside policy. Discussions focused on three specific topics:

1. Strategies used by state agencies administering the MHBG funds to advance ESMI programs.

2. Intervention models followed by the ESMI programs supported by set-aside funds (grantees).

3. States' plans for evaluating grantees.

Organization of the Report

The remainder of the report is organized as follows. First, we describe our case study methodology, including our approach for selecting case study states and holding discussions. Second, we present our case study findings, including: (1) important state-specific context; (2) strategies used to implement the policy by each State Mental Health Authority (SMHA), the agency that administers the MHBG; (3) implementation strategies used by grantees; (4) evaluation strategies used by SMHAs and grantees (including plans for the evaluation of fidelity and outcomes); and (5) barriers and facilitators to implementing ESMI programs. We conclude with a summary of our findings. We note that although each state has a specific name for its SMHA, we use the generic term SMHA throughout for clarity.

Approach for Selecting Case Study States

Selection of the states for the study was done by NIMH and ASPE. NIMH reviewed the 50 states' plans for using the set-aside funds, as proposed in the states' revised FY 2014-FY 2015 MHBG plans, and rated a subset of the states on whether they: (1) had one or more operational CSC or other evidence-based ESMI program as of December 31, 2013, based on NIMH's independent knowledge of a research clinic serving individuals with ESMI, evidence from the Early Assessment and Support Alliance (EASA) or Early Psychosis Prevention and Intervention Center (EPPIC) Directories that an ESMI clinic exists, or evidence that a community clinic was established following involvement in the RAISE study; (2) participated in the NIMH RAISE study; (3) planned to focus on FEP or other ESMI and the treatment model was CSC or an evidence-based variant;²⁴ (e.g., EASA, EPPIC, Portland Identification and Early Referral [PIER], Prevention and Recovery in Early Psychosis [PREP]); and (4) planned to use funds to support already existing programs, establish new programs, develop infrastructure (planning activities, strategic hires, training, etc.), or other unrelated activities in FY 2014-FY 2015.

Twelve states were eventually selected, eight of which were also the focus of a parallel study assessing the impact of the Affordable Care Act on states' use of their MHBG funds. The 12 states were classified into three tiers according to their stage of implementation of ESMI programs, and among states with pre-existing ESMI programs, whether the primary use of the set-aside funds was to expand those or develop new programs. The three tiers are defined as follows (see Figure 1):

• Tier 1: States that had at least one operational ESMI program by December 31, 2013, and are primarily using the set-aside funds to expand pre-existing program(s).

• Tier 2: States that are primarily implementing new ESMI programs.

• Tier 3: States without pre-existing programs that are in the infrastructure-development stage--there are no ESMI programs in operation (i.e., serving clients) as of the end of end of the study's data collection period (June 2015).

FIGURE 1. Map of Case Study States by Implementation Tier
States with the darkest shading represent Tier 1 states; states with the lightest shading represent Tier 3 states.

Given the focus of the study on the impact of the set-aside funds, the Tier 2 states were of greatest interest because the set-aside funds were expected to have the greatest impact in states that did not have prior ESMI programs but had the capacity to establish such programs with the set-aside funds.

The final set of 12 case study states, by tier, is as follows (also see Figure 1):

• Tier 1: California, Connecticut, and New York.
• Tier 2: Idaho, North Carolina, Texas, Virginia, and Wisconsin.
• Tier 3: Colorado, Nevada, Iowa, and Washington.

Approach for Holding Discussions with State Officials and Grantees

Information for the case studies was collected through a series of discussions with key informants, both over the telephone and during site visits. An initial contact was made with the mental health commissioner and mental health planner, who are the key health officials with the SMHA. This initial contact was followed by a phone discussion that covered the state's strategy for use of the set-aside funds. Based on the results of the initial discussion, a decision was made whether to conduct a site visit to the state to observe provider organizations, also referred to as grantees, and hold additional discussions with those parties, or to conduct follow-up discussions with grantees by phone. States in which the set-aside funds had been used to establish newly operational ESMI programs (i.e., Tier 2 states) were prioritized for site visits. States that were still in the planning, training, and infrastructure-development stage and did not have operational ESMI programs (Tier 3) were not directly targeted for site visits. For states with mature ESMI programs (Tier 1) that were not significantly affected by the set-aside funds, we conducted longer telephone discussions. All discussions lasted between one and two hours on average (site visits consisted of multiple discussions lasting 1-2 hours). Details regarding the methods used for each state are provided below and in the Appendix (see State-Specific Case Study Methods and Appendix Table A1).

In collaboration with ASPE, we prioritized the selection of ESMI providers for site visits to achieve a balanced sample with respect to:

1. Diagnostic mix of population served (i.e., FEP-only, any early psychotic disorder, any ESMI), because the empirical evidence is strongest for FEP but the policy allows for the funds to be used in the care of any ESMI.

2. Mix of ESMI models (i.e., CSC, CSC variant, single-component programs, other programs), because the policy is not prescriptive with regard to model despite favoring CSC.

3. Size mix, because size may impact fidelity and sustainability of the program.

4. Provider type mix (e.g., hospital system, CMHC, federally qualified health center), because provider type also could affect fidelity and sustainability of the program.

However, because most states had two ESMI providers at most (and because there was not much within-state variation for those that had more than two regarding diagnostic mix, program type mix, size mix, or provider type mix), the main criterion used for selecting providers was proximity to the state capitol, since each site visit also included in-depth discussions with state officials (see Table 4 for a list of grantees included in the study).

Case Study Results

State-Specific Context

State-Specific Characteristics and Developments That May Influence Implementation of the Set-Aside Policy in the State

We took into account whether the states operated ESMI programs or had been home to RAISE study sites prior to the implementation of the policy, as well as their Medicaid expansion and 1915(i) adoption status (relevant because of the financing opportunities these Medicaid innovations bring with them). In addition, we identified state-specific characteristics and developments that have affected implementation of the policy or have the potential to affect its success in the 12 case study states (see Table 1).

California

California has an extremely decentralized system for distributing its MHBG funds, and the set-aside funds have been channeled into that system. This has two important implications for the use of the set-aside funds. First, decisions about programming are not made at the state level. Rather, the funds, along with much larger amounts of state funds from multiple sources, are directed to 57 county departments of mental health. The counties submit plans for use of mental health resources to the state for approval, but they have broad decisionmaking powers. Second, the set-aside funds are divided among all the counties that request them. While the total number of counties requesting funds was not available, the expectation was that the vast majority would. Because the set-aside funds were divided into relatively small amounts for each county, creating an entirely new program using those funds was not financially feasible for most counties.

Colorado

Although a Denver-based clinic participated in the RAISE study, the ESMI program had been discontinued by the time the set-aside policy was implemented. Moreover, implementation of the policy was delayed due to bureaucratic issues and staff changes and, as a result, the request for proposals was only released in the summer 2015.

Connecticut

The state is the site of two ESMI programs, the Specialized Treatment Early in Psychosis (STEP) program and the Institute of Living (IOL) program, which predate the RAISE study and the set-aside policy (see the section on Strategy for Implementing the Set-Aside Policy).

Idaho

Idaho has one of the lowest rates of mental health spending in the nation, which compounds the challenges of providing mental health services in a state with a large rural population spread over a vast geographic region.

Iowa

Iowa has an approved Medicaid state plan amendment to implement two health home programs;^d the state will use other MHBG funds to fund health home services for adults not eligible for Medicaid.

Nevada

Nevada covers a very large geographic area, but more than 90 percent of its population is concentrated in two urban counties, Washoe County, which includes the city of Reno, and Clark County, which includes the city of Las Vegas. The state's strategy for the set-aside is to start with the urban counties and then build out to the rural areas.

New York

New York benefits from a productive state-academic partnership in the form of the New York State Psychiatric Institute,¹⁷ a state-funded research center affiliated with Columbia University. The institute is home to the Center for Practice Innovations, which under the direction of Lisa Dixon, the principal investigator for the Connection RAISE study, provides technical assistance and resources to states and providers seeking to adopt CSC. The Center for Practice Innovations has developed a training curriculum for the state's slightly modified version of the CSC model, both of which are called OnTrackNY. The state's strategy for using the set-aside funds has been informed by Dixon and her team at the Center for Practice Innovations from the beginning. The team continues to provide consulting and infrastructure support to the expanding OnTrackNY program network.

North Carolina

Outreach and Support Intervention Services (OASIS), the pre-existing ESMI program, was launched in 2005 with the financial support of two foundations that provided a three-year grant. The state began providing MHBG funding to the program after a request was made by a state representative who had heard about OASIS directly from one of its developers. Additionally, the state has partnered with OASIS in a number of mental health initiatives.

Texas

The state decided to limit eligibility for the ESMI programs to uninsured state residents with incomes less than 200 percent of the federal poverty level, thus excluding Medicaid recipients. Because most people under the age of 18 in Texas are covered by Medicaid, this decision also limited the proportion of people ages 18 and under being served by the programs. The SMHA made this decision in consultation with the Health and Human Services commission, the agency under which both the SMHA and the state's Medicaid agency operate. Although respondents reported that a number of factors entered into this decision, a key reason is the expansion of managed care in the state: while managed care organizations (MCOs) pay case rates for a given level of care, they do not have a rate for people with FEP; additionally, MCO rates for complex interventions are much lower than what the state is paying now.

The state has a sophisticated electronic web-based data system, the Clinical Management for Behavioral Health Services (CMBHS), that captures billing and clinical information from all publicly funded programs.

Virginia

Virginia has sought to improve mental health services for transition-age populations since 2005, when the state passed legislation to improve the state's mental health crisis system. The Virginia Tech and Sandy Hook tragedies led to further reforms around civil commitment and temporary detention and prompted former Governor Bob McDonnell to create the Governor's School and Campus Safety Task Force in 2013. The task force developed a number of recommendations, including the enhancement of mental health services for transition-age individuals. Action on this recommendation was accelerated by another tragedy, the assault of Senator Creigh Deeds by his son, Gus, who eventually killed himself while in the midst of a psychotic episode, a tragedy that was widely viewed as the result of his release from a community hospital due to lack of psychiatric beds. These developments led to further mental health reform activity during the Virginia general assembly of January 2014 and, as a result, the 2014 session appropriated general funds to expand services for transition-age youth specifically. The MHBG 5-percent set-aside requirement was established around the same time and the SMHA decided to combine both funding streams to establish multiple ESMI programs in the state. Because the combined funds were sufficient to stand up seven programs with some funds left over, the agency decided to tap into additional MHBG funds to fund an eighth program.

Although the state has not expanded Medicaid, under the direction of the incumbent governor, the Medicaid agency launched the Governor's Access Plan (GAP) in January 2015 to improve access to certain Medicaid-covered behavioral health and primary care services for adults with SMI. (As defined by regulations, SMI determination requires illness of at least one year's duration.) Although eligibility criteria for coverage under the GAP are very strict, including an income of less than 60 percent of the federal poverty level, it is estimated that 20,000 residents would be eligible for Medicaid-covered services through the policy.

Washington

Washington's geographic proximity to Oregon has influenced the state's decisions regarding the design and implementation of the set-aside policy. (Oregon, the state where the EASA model was developed, introduced EASA-type programs for FEP in 2001, and they currently cover more than 90 percent of the population.) Washington has worked closely with Oregon-based experts throughout the policy implementation process.

Wisconsin

No contextual information emerged during discussions that had the potential to influence the impact of the policy.

TABLE 1. State-Specific Characteristics and Developments that May Influence Policy Implementation
Statea	ESMI Program by December 2013b	RAISE Site(s)	Tierc	1915(i) by July 2015	State Characteristics and Developments Affecting the Set-Aside Policy
California	Yes	Yes	1	Yes	Highly decentralized state--set-aside funds were distributed among many counties
Colorado	No	Yes	3	Yes	Bureaucratic issues and staff changes delayed implementation
Connecticut	Yes	Yes	1	Yes	State is the site of 2 ESMI programs predating RAISE and the policy (STEP and IOL)
Idaho	No	No	2	Yes	Low public mental health spending; rural state
Iowa	No	Yes	3	Yes	State has two approved health home programs and will use MHBG funds to cover health home services for uninsured individuals
Nevada	No	No	3	Yes	Rural state
New York	Yes	Yes	1	No	Principal investigator for RAISE study (Connection) is based at Columbia University
North Carolina	Yes	No	2	No	Long-time state support for an ESMI program predating RAISE and the policy (OASIS)
Texas	No	No	2	Yesd	(i) Decision to only offer services to uninsured people excluded Medicaid recipients, including most individuals under 18 years old (ii) State's sophisticated electronic web-based data system
Virginia	No	No	2	No	(i) The Virginia Tech and Sandy Hook tragedies spawned the Governor School and Campus Safety Task Force in 2013, which recommended enhancing mental health services for young individuals. This development and the Deeds tragedy in November 2013 led to a large appropriation of state funds to expand mental health services for transition-age people in January 2014 (ii) The GAP implemented in January 2015 may improve access to health care for uninsured individuals with SMI
Washington	No	No	3	No	Geographic proximity to Oregon, a state with a vibrant FEP tradition
Wisconsin	No	No	2	Yes	N/A
Shading indicates a Medicaid expansion state. See text for method used to make this determination. See text for definition of tiers. However, 1915(i) is almost entirely focused on those being discharged from long-term inpatient facilities: https://www.dshs.state.tx.us/mhsa/hcbs-amh/.

Level of Funding and Activities Funded by the Set-Aside

As shown in Table 2, the dollar amounts associated with the FY 2014 set-aside funds varied widely across states, from $186,765 in Iowa to more than $3 million in New York and California; this variation merely reflected the variable size of the MHBG across states. However, in California, the funds were divided into much smaller portions among the 57 county departments of mental health. Only a handful of states supplemented the set-aside funds with state general revenue (GR) or additional MHBG funds, and this was usually the result of specific legislative action, such as in Virginia, or the availability of state funds for similar purposes, as in California. States also varied in the scope of activities funded through the policy. Most states used the funds to support not only clinical services but also training of the clinical staff, evaluation of the ESMI program, or other activities (e.g., hiring external consultants for program design or development, augmenting other clinical programs, etc.). Only Tier 1 states with existing ESMI providers might have been able to allocate the set-aside funds to existing providers soon after the policy was implemented (FY 2014), but even those states engaged in planning and other infrastructure-development activities before they began disbursing funds to grantees. Hence, the initial set-aside allocation in several states was somewhat larger than the FY 2014 funds (i.e., if the allocation was made in FY 2015, it included the portion of FY 2014 funds not used for infrastructure-development activities and some portion of FY 2015 funds). Thus, some states, such as North Carolina, had more than one year's worth of funds at program launch.

Most Tier 2 and Tier 3 states used the set-aside funds to develop 1-2 ESMI programs; although North Carolina funded three programs, the state developed only two of those; the third program had been in operation since 2005. The large number of ESMI programs developed in Virginia was made possible only by a large infusion of state general revenue funds, $3.5 million for the state's FY 2015 (roughly matching the federal FY 2014) and $4 million for the state's FY 2016 (roughly matching the federal FY 2015), supplemented by a small amount of MHBG funds (see previous section).

The small number of programs allowed concentration of the funds and simplification of implementation and evaluation. New York, a Tier 1 state with more experience and resources, aimed to create a much larger number of new programs, using past experience as a guide. California is an outlier with its large number of programs, the direct result of the state's decision to manage the set-aside funds in the same decentralized fashion that the state manages the MHBG. (We were not able to determine the number of counties that are implementing ESMI programs because many county plans had not been submitted or reviewed at the time of the site visit.)

State Epidemiology of Relevance for ESMI Programs

Table 2 also shows the size of the population in the target age range of the ESMI programs (ages 15-24), and the size of the population in a similar age range (18-24), that is served by the public mental health system in each state. It is important to note that while the prevalence of SMI, as determined by survey assessments of representative samples, is quite similar across states, the proportion of the total population that is served by the public mental health system varies dramatically because of states' demographic characteristics or coverage generosity.

TABLE 2. Size of Set-Aside Funds, Number of Grantees (ESMI Programs), and Target Populations by State
State	Set-Aside Funds ($) FY 2014a	Other Funds Usedb	Number of Grantees to Receive Set-Aside Funds	Persons Ages 18-24 in SMHA (N)c	Persons in SMHA as Proportion of Total State Population Ages 18-24d (%)	NSDUH Estimates of Percentage of SMI, Ages 18-25, Past Year (%)e
California	3,109,278	GR	>10f	72,572	1.81	3.92(CI 3.31-4.65)
Connecticut	341,481	GR	2	8,265	2.39	4.03 (CI 3.12-5.20)
New York	3,090,758	No	10	76,597	3.84	3.94 (CI 3.30-4.69)
Idaho	127,740	No	1	2,422	1.55	5.13 (CI 4.06-6.46)
North Carolina	643,491	No	3	25,786	2.62	4.15 (CI 3.24-5.30)
Texas	1,829,838	No	2	36,650	1.35	3.49 (CI 2.90-4.18)
Virginia	570,327	GR, MHBG	8	10,790	1.30	3.88 (CI 2.99-5.02)
Wisconsin	369,000	Yes	1	11,472	2.05	4.61 (CI 3.63-5.85)
Colorado	341,481	No	1	11,124	2.15	4.41 (CI 3.45-5.61)
Iowa	186,765	No	2	13,027	4.08	4.27 (CI 3.31-5.48)
Nevada	227,500	No	1	3,077	1.21	4.37 (CI 3.37-5.66)
Washington	521,452	No	1	14,568	2.18	4.61 (CI 3.60-5.87)
NOTES: SOURCE: FY2014 Revised MHBG Plan. For program implementation--this is limited to substantial and direct financial commitment from other state sources, including GRs and MHBG. It does not include funding for services that might also be provided to FEP program clients or reimbursements for care. SOURCE: MHBG Unduplicated Persons Served Count by Age Group with Gender and Race Breakdown, Report Year: 2015, Application for SAMHSA Block Grants and PATH Grants. Based on U.S. Census estimate of 2013 state population ages 18-24. Annual Estimates of the Resident Population for Selected Age Groups by Sex for the United States, States, Counties, and Puerto Rico Commonwealth and Municipios: April 1, 2010, to July 1, 2014, U.S. Census Bureau, Population Division. SOURCE: Table 23: Serious Mental Illness in the Past Year, by Age Group and State: Percentages, Annual Averages Based on 2012 and 2013 NSDUHs. National Survey on Drug Use and Health: Model-Based Prevalence Estimates (50 States and the District of Columbia), Year 2012-2013. Not available. It remains unclear how many ESMI programs are supported by the set-aside because implementation is at the county level and we were not able to survey all the counties.

Strategy for Implementing the Set-Aside Policy

As shown in Table 3, states vary in their strategies for implementing the set-aside policy and in the models selected to serve individuals with ESMI. The clinical target population identified by states varies from only individuals with FEP (early, non-organic, non-affective, psychotic disorders), to those with any early psychotic disorder, to a still broader category of individuals with any ESMI. ESMI programs in most states serve youth and young adults, with the majority of programs serving individuals within or near the age range of 15-25. Maximum or expected program caseloads vary across states, and span from 15 to 151 individuals per team.

Implementation Strategy as Designed by the SMHA, the State Agency Administering the MHBG

We next explore the SMHAs' strategies for implementing the set-aside policy (see Table 3). We describe whether the strategy involved ESMI program expansion or development, and the number of ESMI programs (grantees) selected along with the states' rationales for their decisions. We also describe the states in terms of how prescriptive they were with regard to the model to be implemented (i.e., CSC versus the grantee's choice), target population (i.e., FEP-only, any early psychotic disorders, any ESMI, or the grantee's choice), training curriculum (i.e., OnTrackNY, NAVIGATE, or the grantee's choice), and uses of the set-aside funds (i.e., clinical services, training, program evaluation, other). We describe inclusion and exclusion criteria only if the SMHA was involved in their selection. Lastly, we describe the states' financing strategies with regard to the operations of ESMI programs for states that had a clearly delineated strategy; we note that the main difference among states with a clear strategy was whether there was an explicit expectation that the set-aside funds would be supplemented by third-party reimbursement when services were rendered to insured clients.

TABLE 3. State Strategy and Grantee Description
State	State Strategya (Expand and/or Develop)	Grantee(s) Contacted	Model Selected:b CSC, Evidence-Based Variant, Other	Target Population:c FEP, Any Psychotic Disorder, Any ESMI	Age Criteria	Maximum or Expected Caseload per Program/Team
TIER 1
California	Expand and develop	Alameda	PREP	FEP	16-24	60
		Fresno's FOT Team	TIP	Any early psychotic disorder	16-28	30
		Imperial	PIER	Any early psychotic disorder	15-25	15
		San Luis Obispo	Other	Any person with mental illness experiencing a mental health crisis	18-22 (college students)	4-7 per week; 200 per year
Connecticut	Expand	IOL	Other, Influenced by EPPIC	Any ESMI	16-24	75 current caseload; ~250 individuals per year
		STEP	STEP	FEP	16-35	Caseload currently ranges 100-200 patients per year
New York	Expand and develop	Northern Rivers Family Services	CSC	FEP	16-30	30
TIER 2
Idaho		St. Luke's Regional Health Center	CSC	Any early psychotic disorder	18-40	30
North Carolina		OASIS (including OASIS Carrboro and Wake STEP)	CSC	Any early psychotic disorder	16-30, with flexibility	Carrboro site: 151; Wake STEP site: ~9
Texas		Enhanced ePEP (Dallas Metrocare)	CSC	Any early psychotic disorder	15-30	30
		MHMRA of Harris County	CSC	Any early psychotic disorder	15-30	30
Virginia		Alexandria CSB	CSC	Any early psychotic disorder	16-25, with flexibility	30
		Fairfax-Falls Church CSB	CSC	Any early psychotic disorder	16-25, with flexibility	60
		Highlands Community Services	CSC	Primary target: FEP; Secondary target: individuals with second psychotic episode or with early bipolar disorder	16-25, with flexibility	Target number of enrolled clients: 12-15d
		Western Tidewater CSB	CSC	Primary target: any early psychotic disorder; Secondary target: any ESMI	16-25, with flexibility	30
Wisconsin		JMHC	CSC	FEP	15-25	30
TIER 3
Colorado		Jefferson Center for Mental Health	CSC	Any early psychotic disorder	15-25	No target caseload determined yet
Iowa		CMHC, Des Moines	CSC	FEP	15-30	Estimated statewide target population is 927, but no team caseload target yet
		CMHC, Cedar Rapids	CSC	FEP	15-30	Estimated statewide target population is 927, but no team caseload target yet
Nevada		The Children's Cabinet of Northern Nevada	EASA	FEP	15-25	30
Washington		Central Washington Comprehensive Mental Health	EASA	FEP	15-25	N/A
We describe only for Tier 1 states because by definition, all Tier 2 and Tier 3 states will develop new programs. For the purposes of this table, we describe RAISE models including OnTrackNY as CSC; evidence-based variants include EASA, EPPIC, PREP, PIER, and TIP among others (for a detailed description of evidence-based models for individuals with early psychosis, see: http://onlinelibrary.wiley.com/doi/10.1111/eip.12132/full). Any early psychotic disorder includes FEP and early affective psychotic disorder (bipolar disorder, major depressive disorder with psychosis); any ESMI includes FEP, early affective psychotic disorder, and non-psychotic SMI such as obsessive-compulsive disorder. Substance-induced psychosis is excluded unless specified. Low target is related to the program's rural setting.

SMHA Strategy: Tier 1 States

California

As previously noted, California did not have a centrally determined strategy for implementing the set-aside policy; to the contrary, strategies were determined by each of the counties that submitted plans to use their potential allocation of set-aside funds. The state was not prescriptive in its implementation of the policy beyond requiring that use of the funds meet SAMHSA criteria. We were unable to determine what proportion of counties had their set-aside plans approved by the state and received funds to expand or develop ESMI programs. Of the four-county sample with which we held discussions,^e two counties (Alameda and Fresno) applied their allocation of set-aside funding toward expansion of existing county-run ESMI programs, while the other two (Imperial and San Luis Obispo) pursued the development of new programs.

Connecticut

Connecticut's strategy for implementing the set-aside policy involved expanding the state's two existing ESMI programs, the STEP Program (based at Yale University, New Haven), and the IOL Program (in Hartford), because the amount of funding was only sufficient to expand pre-existing FEP programs. The state was not too prescriptive in its implementation of the policy, limiting itself to suggesting new ways for the existing programs to expand. We note that the Department of Children and Families (DCF), which resides within a separate state agency, receives 30 percent of the MHBG funds and also received a similar proportion of the set-aside funds. DCF decided to use its portion of the funding to enhance a program focusing on trauma identification and intervention.

New York

New York's strategy for implementing the set-aside policy was to draw on the experience of the existing RAISE programs and expertise and on the empirical foundation of the CSC model and its implementation at the Center for Practice Innovations in New York City, to develop a network of ESMI programs across the state. The state was highly prescriptive in its implementation of the policy, including choice of model, target population, training curriculum, and uses of the funds. The state is implementing the OnTrackNY model, an adaptation of the CSC model used in the RAISE study, to offer services to state residents with FEP. Close to 90 percent of the set-aside funds were used for the four existing RAISE programs in the New York City region and several new FEP programs. The existing programs are being lightly modified to better adhere to the OnTrackNY model. The remaining funds are being used to support the Core Team, which is responsible for providing statewide support for the initiative, including training (which follows the OnTrackNY curriculum) and developing the evaluation infrastructure. Although the state estimates that 30 OnTrackNY teams ultimately will be needed to cover the entire state (based on an estimate of 20-30 cases per 100,000 population), the current plan is to establish six new OnTrackNY teams across the rest of the state. By the time of our site visit, there were six teams in infrastructure-development stage. The new OnTrackNY team based in Albany, which we visited, is administered by Northern Rivers, a multispecialty human services agency with a focus on services for children.

Inclusion/Exclusion Criteria. In keeping with the OnTrackNY model, the state determined that grantees would target individuals with FEP ages 16-30 within two years of their diagnosis.

Financing Strategy. The new programs will be fully funded through the set-aside funds for the first six months. During this time, Northern Rivers, in collaboration with the state, will examine the extent to which the services are billable to Medicaid or other insurance carriers and develop a sustainability model.

SMHA Strategy: Tier 2 States

Idaho

Idaho's strategy for implementing the set-aside policy was to develop new ESMI services. The state selected St. Luke's Regional Health Center in Magic Valley-Twin Falls as the sole grantee. Using the same mechanism through which the SMHA enters into contractual relationships with MHBG grantees, the state awarded the contract for the ESMI program to St. Luke's without competition. This was based on the state's assessment that St. Luke's had the capacity, as well as the population needs and resources available within the region, to implement an ESMI program successfully. Idaho was highly prescriptive in terms of choice of model, method to estimate the target population, training curriculum, and uses of the funds. The state decided to implement the CSC model. The SMHA planned to use the set-aside funds to contract for a needs assessment to identify and quantify the potential target population for the program, fund a nine-month pilot CSC program, and fund an evidence-based training program, data analysis, and outcomes reports.

Financing Strategy. The state is delivering the set-aside funds in fixed monthly amounts to compensate for services rendered. While some clients may be more costly than others, St. Luke's is responsible for covering the cost of services with the monthly set-aside allocations. Additionally, the state expects that St. Luke's will leverage clients' insurance for reimbursable services, and will not turn individuals away based on financial need.

North Carolina^f

North Carolina has had an early intervention program for people with early psychosis based on the EPPIC model called OASIS since 2005. Although its developers stated that the model shares some critical components with CSC, they noted that OASIS places a strong emphasis on general health and wellness as a fundamental aspect to the recovery process. Furthermore, the program does not discharge patients, thus carrying a large caseload comprising clients in varying stages of acuity and recovery. The original program located in Carrboro in Orange County, was initially funded with a three-year grant from KB Reynolds and the Duke Endowment, but it has received MHBG funding over the years. North Carolina's strategy for implementing the set-aside policy was to provide further support to the OASIS program, as well as develop two new ESMI programs. The SMHA issued an invitation to apply for the set-aside funds, expecting that the state would contract with OASIS and additional providers. After reviewing the proposals, the SMHA selected two new providers: a newly established OASIS site based in Raleigh in Wake County (the Wake Schizophrenia Treatment and Evaluation Program, or Wake STEP, managed by the Alliance Behavioral Healthcare MCO), and a provider called RHA Behavioral Health Services based in Wilmington in New Hanover County (managed by the CoastalCare MCO). In selecting the new clinical sites, the state evaluated the size of youth population and tried to not focus exclusively on urban areas. Because OASIS has a good track record of outreach to other counties, they expected a broad geographic reach for the two OASIS programs. Although the Wilmington area is quite populous and there is a university nearby, their expectation is that RHA Behavioral Health Services would also reach the rural areas in that part of the state.

The state was only moderately prescriptive in its implementation of the policy. North Carolina decided to implement the CSC model and, as part of this decision, the state provided funding to the original OASIS program to help it become more CSC-like. North Carolina's decision to adopt CSC as opposed to OASIS was influenced by the state's understanding that the models were quite similar and the desire to adhere to the SAMHSA guidance as closely as possible. The state distributed more or less equal allocations to the three programs, but the funds allocated to the original OASIS-Carrboro Program included funds for technical assistance and the evaluation component (the state recommended several data elements for consideration but was not otherwise prescriptive). Funds to the OASIS-Carrboro program were also used to hire staff, some of whom had been previously available because of their university connections but in an inconsistent fashion. The state was not prescriptive in terms of target population (i.e., whether FEP or any early psychotic disorder) or regarding training curriculum--although OASIS has developed its own training curriculum with set-aside funding, the RHA Behavioral Health Services site has also used OnTrackNY training resources.

Financing Strategy. The portion of the funds allocated to clinical services will be primarily used to support staff salaries; the balance of the funds, approximately 25 percent, will be used for program supplies, transportation, logistics, and covering non-reimbursable services and the uninsured/underinsured. While the SMHA anticipates that the grantees will identify revenue sources and work toward obtaining reimbursement from third-party payers, state respondents reported that their priority at the moment is for providers "to work on getting the practice model in place firmly...getting the administrative components up and running and the staff trained." The original OASIS program captures private insurance for some of their services, as illustrated by a breakdown of recent funding streams: 60 percent private, 23 percent Medicaid, 2 percent Medicare, and 15 percent with no insurance.

Texas

Texas's strategy for implementing the set-aside policy was to develop new ESMI services. The state was highly prescriptive in its implementation of the policy, including choice of model, target population, training curriculum, and uses of the funds. Texas is implementing the CSC model to offer services to individuals with any early psychotic disorder. The state selected two providers as grantees, with the intention of funding two teams per grantee: the Mental Health and Mental Retardation Authority (MHMRA) of Harris County, a Houston-based Local Mental Health Authority (LMHA), and Metrocare Pathways Clinic, a Dallas-based North Texas Behavioral Health Authority (NTBHA) provider.^g The SMHA used the following criteria to select grantees: (1) location in urban areas to maximize the chances of referrals (citing the recommendation in the RAISE literature that a fully dedicated team should cover a population of about 500,000 to be sustainable); (2) close proximity to academic institutions capable of leading the evaluation component; (3) ability to compare the CSC implementation process for an LMHA-based provider (operating through a regular state performance contract) and a provider operating under ValueOptions, the carve-out managing care for NTBHA; and (4) previous experience providing ESMI services for transition-age youth (Metrocare-Dallas). The state specified target caseloads (30 clients per team) and staffing (six-member teams, including a peer specialist). The SMHA contracted with the Center for Practice Innovations to provide OnTrackNY training to the new teams using set-aside funds, and required that the grantees use the set-aside funds to subcontract with academic institutions to evaluate the newly developed programs.

Inclusion/Exclusion Criteria. The state required that the grantees target individuals with any early psychotic disorder ages 15-30 within two years of their diagnosis.^h As previously mentioned, Texas also determined that the set-aside funded services would only be available to uninsured state residents with incomes less than 200 percent of the federal poverty level.

Financing Strategy. The state had the expectation that the grantees would tap into the organizations "with large revenue streams" that they are nested within (i.e., MHMRA of Harris County and Metrocare Pathways Clinic) for financial assistance with staffing, thus permitting them to use a larger portion of the set-aside funds for their operations. Although grantees receive one-twelfth of the budget at the end of each month to ensure they can operate, they receive the balance only if they deliver a predetermined number of treatment hours, a strategy designed to promote productivity ("per diem process"). Although the SMHA had interpreted from the SAMHSA guidance that the set-aside funds could not be used to cover outreach activities and as a result, they are not covered by the "per diem process," respondents thought that the generosity of the per diem rate and the fact that the teams were not at capacity yet would provide adequate incentives for the teams to engage in outreach and client recruitment activities.

Virginia

Virginia's strategy for implementing the set-aside policy was to develop new ESMI services. In July 2014, the SMHA released a request for proposals to the state's 40 Community Service Boards (CSBs)ⁱ stating that "the FY 2015 state budget [~FY 2014 federal budget] and the federal MHBG include funding to support the development and expansion of services for young adults ages 16-25 experiencing serious behavioral health conditions, including substance use/abuse and the initial onset of psychosis. Through a combination of these funds, [the SMHA] will have approximately $4 million available in FY 2015 and beyond to support multiple awards for demonstration initiatives to develop services for this population."^j The state was moderately prescriptive in its implementation of the policy, expressing a preference for the CSC model in its request for proposals, and identifying a broadly defined target population that includes people with non-psychotic disorders. While requiring that the grantees "measure, monitor and report fidelity to their chosen modality and outcomes of services provided," the state was not prescriptive about the evaluation plan and it did not allocate separate set-aside funds to the evaluation. The state successfully applied for additional SAMHSA technical assistance funds to make OnTrackNY trainings provided by the Center for Practice Innovations available to grantees.

Fourteen of the 18 applications from five of the seven regions in the state proposed a CSC model. Because the review committee wanted to ensure adequate geographic representation of ESMI services across the state, the top-scoring application from each of the five regions was selected along with the three top-scoring CSBs from the remainder of the applications, thus selecting eight grantees.

Financing Strategy. Virginia did not have an expectation that the grantees would need to bill third-party payers or use additional funding sources to operate the programs. In its request for proposals, the state did ask prospective grantees to "describe any existing funds or in-kind support (that would be allocated) to the proposed services and any other anticipated revenues (e.g., Medicaid) that would be generated," which several of the successful applicants did.

Wisconsin

Wisconsin's strategy for implementing the set-aside policy was to develop new ESMI services. The SMHA determined that the annual set-aside fund allocation for Wisconsin would be sufficient to support one ESMI program providing services for a population of approximately 500,000-525,000, based on recommendations from the RAISE literature. The state released a request for proposals in August 2014 with the intention to launch the winning contract on October 1, 2014, and start service delivery in early 2015. The SMHA required that applicants "have, or show they can develop...identified expertise in providing care to youth who are experiencing psychotic illness; wellness, recovery, and resilience orientation; access to inpatient hospital care; linkages with community resources and outreach capabilities; strong psychiatric supervision and clinical leadership; commitment to hiring individuals with lived experience; and ability to provide data."^k Furthermore, the SMHA required that applicants have a certified psychosocial rehabilitation program "to serve as a foundation for the CSC program."^l The state selected Journey Mental Health Center (JMHC), a large and established provider located in an urban region (Madison in Dane County) as the sole grantee.

The state was highly prescriptive in its implementation of the policy, including choice of model, target population, training curriculum, and uses of the funds. The state decided to implement the CSC model to serve individuals with FEP. The program was expected to maintain a caseload size "of 30 clients or less to ensure adequate time to maintain fidelity to the model services, develop relationships, and provide outreach to the participants," and provide time-limited services (2-5 years). The state decided that the set-aside funds would be used solely to implement the CSC. The SMHA determined that oversight of program fidelity would be done by the agency's contract manager, whose salary is paid for by the larger MHBG. The outcomes evaluation component will be designed and overseen by a researcher affiliated with the University of Wisconsin-Madison who has volunteered to undertake this activity free of charge. Using SAMHSA funds, the state recently contracted with Center for Practice Innovations to provide technical assistance to both the grantee and a Milwaukee-based provider that had failed in its bid for the funds but decided to set up a CSC program using local funds.

Inclusion/Exclusion Criteria. The SMHA determined that the grantee would target individuals with FEP ages 15-25 within three years of symptom onset.

Financing Strategy. The SMHA expects that the grantee will bill private insurance and Medicaid for all billable services, including psychosocial rehabilitation services, to ensure program sustainability.

SMHA Strategy: Tier 3 States

Colorado^m

The planning for the implementation of set-aside policy was conducted with input from the state's Behavioral Health Planning and Advisory Council and Dr. Lisa Dixon and her team at the Center for Practice Innovations. The Center for Practice Innovations has been contracted to provide consultation and technical assistance for implementation and evaluation of the program. The SMHA decided to implement the CSC model and fund one grantee to be selected through a competitive bidding process. The state's expectation was that the experience from the funded program would inform the future expansion of the model, with the goal of better meeting the needs of transition-age youth across the state. The SMHA had anticipated receiving responses from CMHCs, other specialty mental health clinics, and possibly the University of Colorado. The request for proposals stated that the successful grantee would implement a high-fidelity CSC program with the goal of improving outcomes for youth with early psychosis, including affective disorders (i.e., any early psychotic disorder). The state required that applicants commit to adhering to existing regulations if they are not already an approved CMHC or a specialty clinic. The SMHA also expressed a preference for proposals that would build on existing services targeting youth ages 15-25 with SMI. Colorado ultimately selected Jefferson Center for Mental Health as the state's sole grantee. The provider is located just west of Denver in Jefferson County, one of the most populous counties in the state. The state envisions that the program will also serve two neighboring mountain counties, slightly more rural, to meet caseload requirements.

Thus far, Colorado has been moderately prescriptive in its implementation of the policy, having selected both the model and the target population.

Inclusion/Exclusion Criteria. The SMHA determined that the grantee would target individuals with any early psychotic disorder ages 15-25 within three years of symptom onset.

Iowa

Iowa's initial strategy for implementing the set-aside policy was to allocate all the funding to an existing RAISE study site with the goal of expanding the program. However, the RAISE team dissolved sometime after the policy was implemented, so the SMHA decided to develop new ESMI programs. The state decided to implement the CSC model to serve individuals with FEP, and fund two grantees whose training could be combined to achieve economies of scale. The SMHA only received two applications in response to their request for proposals. Potentially eligible providers in rural areas relayed to the SMHA that they would not have the resources to develop and deliver CSC services with adequate fidelity. Contracts were awarded in April 2015 to two CMHCs. The state has been highly prescriptive in its implementation of the policy. The state selected the model, target population, and training curriculum (NAVIGATE) for the CSC programs.

Inclusion/Exclusion Criteria. The SMHA determined that the grantees would target youth and young adults ages 15-30 experiencing FEP.

Nevada

Nevada's strategy for implementing the set-aside policy was to develop a new ESMI program modeled after the EASA model. During the planning phase, the state has worked closely with its chosen provider, The Children's Cabinet, a large child social services provider in Reno, and consultants from Oregon's EASA/FEP program to develop a detailed list of services that will be included in the program, a listing of team members and their roles and responsibilities, and an organizational chart showing the institutional location of each team member. The program is scheduled to start operating during the second half of 2015 in Reno and then expand to Las Vegas at a later point in time, after gaining initial implementation experience. The Children's Cabinet is the largest child social services provider in the Reno area and was well positioned to establish working relationships with the full range of providers required by a FEP program.

Inclusion/Exclusion Criteria. The SMHA determined that the grantee would target individuals with FEP ages 15-25 who were within 18 months of symptom onset. Individuals will be excluded if they have received treatment for psychotic symptoms for more than three months; have a significant medical condition that prevents participation in services; or are incarcerated. These criteria will be followed during the implementation of the Reno program and potentially adjusted according to that initial experience.

Financing Strategy. The state expects that state-funded clinical staff will play a large role in the FEP program--this strategy would allow the program to have access to medical staff without having to support them financially. The state also expects that the FEP program will actively seek reimbursement from third-party payers.

Washingtonⁿ

Washington's strategy for implementing the set-aside policy was to allocate the bulk of the funding^o to develop a new ESMI program modeled after the EASA model and conceived as a demonstration project that will inform future expansion of ESMI services to other regions in the state. The state has worked closely with EASA experts based in Oregon^p throughout the program development process.

The SMHA selected Central Washington Comprehensive Mental Health, a large multispecialty human services provider, as its sole grantee; its ESMI program is called The New Journeys program. The provider is located in a relatively rural area of the state, Yakima County, with a population size of 250,000. There were several reasons behind their choice of provider:

1. The provider already offers the full range of services that are needed to implement a CSC program--including a broad range of health care professionals, many of whom are bilingual, and an intensive case management program for SMI individuals (Wraparound with Intensive Services); moreover, services are integrated into a single electronic health record (EHR) system that will greatly facilitate care coordination.

2. The agency has strong community connections, including connections with law enforcement, that will facilitate outreach and naturally configure a network of providers to support program clients.

3. The agency has experience billing both Medicaid and private insurers for services, whereas many agencies would require extensive technical support to develop that capacity, which is critical to the long-term sustainability of the program.

The state has been highly prescriptive in its implementation of the policy, including choice of model, target population, training curriculum, and uses of the funds. The SMHA selected the EASA model based on its assessment that the model has proven to be successful in improving outcomes for youth and young adults statewide in Oregon. The state determined that, in keeping with the EASA model, the target population would be restricted to individuals with FEP. Moreover, it was determined that the program would be time-limited, with two years for most individuals, and a possibility of extending the program to three years for those in need. Washington has contracts in place for clinical training, educational presentations for providers on FEP programs, and consultation for implementation of the EASA model.

Inclusion/Exclusion Criteria. The SMHA determined that the grantee would target young individuals with FEP.^q While the program is located in Yakima County, once it has established a good foundation in that county, the catchment area will be extended to surrounding counties.

Implementation Strategy as Designed by the ESMI Programs

Next, we describe in detail the ESMI programs that we held discussions with (see Table 3). For states where programs had ample latitude in terms of model, training curriculum, target population, or uses of the funds, we describe those domains and describe program status--including whether operational, and current caseload, if so. For states where we were able to collect information on partnerships and collaborations, we also discuss this component of program implementation. Programs' partnerships and collaborations are important because they provide a valuable mechanism for outreach to potential referral sources,^r which is also described in this section whenever the information was captured. (Partnerships and collaborations are also important for design and implementation of the evaluation strategy, which is described in the next section.)

Program Strategy: Tier 1 States

California

Alameda County

Alameda County used the set-aside funds to enhance a pre-existing FEP program based on the PREP model²⁴ and located within a larger Transitional Age Youth (TAY) program. The PREP Program assists up to 60 youth and young adults with recent onset of non-affective psychosis by offering direct and accessible clinical services. Concerns about sustainability of the set-aside funding led county officials to use the funds for training and capacity-building within the PREP program, thought to be very similar in content to CSC, rather than change the model. The training is provided by the Felton Institute, a private non-profit social service provider and developer of the PREP program based in San Francisco, and it focuses on strengthening the clinical skills needed to successfully implement the model. The training covers methods for early identification of psychosis, differential diagnosis of SMIs, and Cognitive Behavioral Therapy for Psychosis (CBTp). The training consists of initial sessions held at the institute and follow-up supervision and feedback provided over six months following the in-person sessions.

Partnerships and Collaborations. The program works closely with clinical partners at University of California-San Francisco (UCSF) and training partners at the Felton Institute. The program is staffed with UCSF-based psychiatrists and the Felton Institute provides training to program staff in the clinical components of the model and consultation to the program regarding evaluation methods.

Fresno County

Fresno County used the set-aside funds to enhance the First Onset Psychosis Team (FOT), a pre-existing program that had been in operation since 2010 supported with funds from the Mental Health Services Act^s and Medicaid reimbursements. The FOT carries a caseload of about 30 clients between 16 and 28 years of age with any early psychotic disorders. The program consists of six clinicians and four case managers who divide their time between this program and a more general program for at-risk TAY. The FOT uses a Transitions to Independence Process (TIP) model,^{25, t} described as CSC-like, to provide a CBTp-based treatment that also includes psychopharmacology. The set-aside funds are being used to enhance the program and augment it with CSC components. Specifically, the funds support: (1) CBTp training provided by the Felton Institute and targeted to team psychologists; (2) peer support training in the TIP model provided by a large provider of community mental health services located in Long Beach (STARS Behavioral Health Group); and (3) consultant-led training in the CSC model.

Partnerships and Collaborations. The program had prior partnerships with hospitals, crisis centers, and emergency rooms in the county, which are currently their main referral source.

Outreach to Potential Referral Sources. Client recruitment occurs in multiple settings across the counties, through outreach by the team and referrals from collaborating providers. The team has positive relationships with hospitals, crisis centers, and emergency departments in the county and those sites have been the largest sources of patients to date. Team members visit potential clients during psychiatric hospitalizations. Although the program is about to reach capacity, it is extending the referral network beyond health care institutions to local schools and colleges.

Imperial County

Imperial County used the set-aside funds to augment the capability of its existing youth services program, called Transitional Engagement Supportive Services (TESS), with the goal of addressing the needs of adolescents and young adults experiencing ESMI. The TESS program provides a host of supports to individuals being discharged from the hospital or other crisis intervention services for mental illness to assist with their reintegration into the community.^u

The set-aside funds will be used to strengthen a specialized component of the TESS program that will connect adults ages 15-25 who are at risk for or currently experiencing the first episode of any psychotic disorder with the county's existing Full Service Partnership (FSP) program.^v The county described this strategy as an efficient way to improve services for this population because the TESS program already provides services to young adults.

To provide the connection to the FSP program, the county adopted the PIER model,^w which was recommended by its creator, William McFarlane, and had been successfully implemented in other California counties. County officials decided to augment the PIER model with a family involvement component they thought would be a good fit for their population, a large proportion of which is Latino. The PIER model will be implemented largely by tapping into existing FSP resources and using the set-aside funds to support additional clinical staff time. The set-aside funds will enable the PIER program to screen for recent onset psychosis and expand outreach capacity through presentations in schools and education about psychosis for law enforcement. Respondents described that part of the PIER model includes a preliminary phase (approximately 3-4 months) of outreach and community education, followed by initiating program referrals. The program completed its outreach phase and identified three clients at the time of our discussion in April 2015. The county anticipates that the program will ultimately serve 15 clients in the set-aside target population. The program has been operational since January 2015.

Outreach to Potential Referral Sources. Program staff plan to provide presentations to law enforcement and at schools to improve awareness of psychosis and as a client recruitment strategy.

San Luis Obispo County

Given that the county did not have a pre-existing ESMI program, coupled with the small size of the set-aside funding allocation, the county decided to use the funds to support the development of a mental health crisis intervention program within the residence halls of the California Polytechnic State University (Cal Poly), a university with a large and growing residential student body. The plan was developed in collaboration with Cal Poly's health staff. The program will work with the county's mobile crisis team to link individuals to more-intensive services, such as hospitalization. The county also hopes to introduce a student mental health liaison at the university using Mental Health Service Act funds, but that decision has not been finalized. The target population includes individuals with any mental illness experiencing a mental health crisis and those with co-occurring substance use disorders. At the time of our telephone discussions, the overall structure of the program had been finalized and the county was about to start advertising positions with the goal of being fully staffed at the opening of the 2015 fall semester.^x

Partnerships and Collaborations. The program has a strong partnership with Cal Poly's administration and police department. Additionally, the program coordinates with the county's mobile crisis provider, which is available 24 hours a day, seven days a week, for immediate response including involuntary psychiatric holds.

Connecticut

Institute of Living

The IOL program, based in Hartford, was launched in 2004 as a "grass-roots" effort designed to serve individuals ages 16-24 with ESMI, by treating both the individual and the family and providing cognitive remediation with a heavy emphasis on family education and support over time. Services were developed in conjunction with the International Early Psychosis Association and included influences from Australia's EPPIC program²⁴ and research based in Norway; additionally, William McFarlane was brought in as a consultant. The program provides services to both individuals and their families at no cost. The program provides a range of services for young adults, including an intensive outpatient program, a dual-diagnosis program, and a Lesbian, Gay, Bisexual, Transgender, and Queer program. At the request of the SMHA, the IOL program used the set-aside funds to augment its vocational services (including job readiness and payment for persons taking part in supervised work assignments), outreach, and peer services. Thus, the set-aside funds are being used to expand ("enrich," as described by a respondent) the services that were already being provided by the IOL program.

While the IOL program in 2004 consisted of 12 hours of staff time per month, the program now has 12 full-time staff supported through MHBG funds and from billing for intensive outpatient program services and other services. The program's current caseload is approximately 75 individuals. The majority of these clients are receiving early intervention services, which consist of three encounters a week per person. The program serves 150-200 clients per year.

Financing Strategy. In addition to actively billing third-party payers for billable services and the recent infusion of set-aside funds, the program taps into several funding sources. For example, funding for client engagement services is provided by private, state, and MHBG funding, and costs associated with family-oriented services are covered by a foundation.

Specialized Treatment Early in Psychosis

The STEP program, affiliated with Yale University, New Haven, was set up in the early 1960s to help transition people from state mental hospitals back into the community,²⁴ following similar models that were being implemented in the United Kingdom, Australia, and Northern Europe. The model has undergone revisions, most recently in 2013, mainly related to the psychosocial interventions delivered as part of the program. The program serves individuals with FEP ages 16-35. The MHBG set-aside funds are being used to enhance the vocational services component of the STEP program.

The clinician-patient ratio is 1:30 or 1:40, with mostly office-based services mainly due to lack of outreach-enabling resources. The maximum caseload at the STEP Program has fluctuated since its inception, and currently ranges from 100 to 200 patients per year.

Financing Strategy. The version of the model established in 2006 lacked specific funding sources, which is notable because similar programs had only existed in countries with universal access to health care.

Outreach to Potential Referral Sources. Approximately 50 percent of the referrals come from a private non-profit inpatient psychiatric hospital and more than a third of the referrals are from triage services at the Connecticut Mental Health Center. Other referral sources are a walk-in clinic, private providers, and self-referrals following an advertising campaign.

Partnerships and Collaborations (IOL and STEP). Both programs reported they rely on the LMHA for certain components of their ESMI models (crisis services, intensive case management).

New York

At the time of our site visit, the new programs were in the very early stages of implementation, either still in contract negotiations with the state or just beginning to hire staff. However, Northern Rivers Family Services, the OnTrackNY provider based in Albany with which we held discussions, is already receiving phone calls with potential referrals from the community. Although the program is in its very early stages and several months from enrolling its first client, Northern Rivers staff has begun serving some of the newly referred individuals in their existing mental health clinic until the OnTrackNY program is operational. The program will draw from residents of four surrounding counties (Albany, Schenectady, Rensselaer, and Saratoga) to build up its target caseload of 30 clients.

Partnerships and Collaborations. Northern Rivers has been developing methods for evaluating the FEP program with experts affiliated with Columbia University.

Outreach to Potential Referral Sources. In keeping with the OnTrackNY model, program staff plans to develop close relationships with providers in the region to establish a referral network. The plan is to focus initially on emergency departments and psychiatric inpatient units where young people experiencing FEP are likely to be found. Once the program is more-established, the plan is to expand outreach to medical practices, schools, probation officers, peer-run services (such as People Inc., a local peer-run service agency), school districts, and community organizations in an effort to identify youth in distress and to educate potential referral sources about the recovery-based approach.

Program Strategy: Tier 2 States

Idaho

St. Luke's Regional Health Center, the sole grantee, selected the RAISE (CSC) model based on a review of the evidence and the grantee's assessment of what was feasible to implement in the state. The decision was also helped by participation in webinars provided by experts based in Oregon, Maryland, and New York, as well as consultation with the Center for Practice Innovations. Respondents reported that St. Luke's implemented the CSC model in a way that was largely consistent with OnTrackNY; the program serves individuals with any early psychotic disorder. As above, the state had prescribed that the set-aside funds would be used to fund a needs assessment (which included a chart review), CSC trainings for staff, implementation of a nine-month clinical pilot program serving the population identified in the needs assessment, and an outcomes evaluation.

The program targets individuals ages 18-40 with any early psychotic disorder within two years of their diagnosis. The higher age cap resulted from the needs assessment, which convinced the program developers that such a modification was needed to ensure sufficient number of program participants. The program serves an eight-county region, the catchment area of the local Department of Health and Welfare. The area is approximately 12,000 square miles and including a total population of approximately 185,000 people. At the time of our site visit (April 2015), the fully staffed program had been in operation for approximately two months.

Partnerships and Collaborations. St. Luke's is working with local partners to enable client access to transportation, housing services, and vocational rehabilitation.

Outreach to Potential Referral Sources. The program conducts outreach activities at the inpatient unit of the local psychiatric adult hospital and outpatient clinical settings, including St. Luke's own outpatient behavioral health clinic.

North Carolina

While both OASIS programs target individuals ages 16-30 with any early psychotic disorders, they have flexibility on both ends of the age spectrum and have an assertive policy of not excluding those with co-occurring conditions, such as developmental disabilities, borderline personality disorder, or substance use disorder. The catchment area extends to within 1.5 hours driving distance of either site location.

The OASIS program based in Carrboro in Orange County, in operation since 2005, receives 4-5 referrals monthly. While approximately 20 percent of the clients receive intensive treatment services at a given time, a majority of clients have stabilized and are not currently receiving intensive services. Approximately 10 percent of the client population has more-chronic symptoms. The program's current caseload is 151 clients, but more than three-quarters of them (N=113) have been in the program for more than two years, receiving services at the Carrboro site. Although many of these clients would not be considered to have an early psychotic disorder, as previously noted (see State Strategy, Tier 2 States), the OASIS model differs from CSC in that the program does not discharge patients. The OASIS Program based in Raleigh in Wake County (Wake STEP) has been in operation for a shorter period of time but has expanded rapidly and is now receiving many referrals. As of August 2015, the caseload of the Raleigh site was approximately nine clients.

Partnerships and Collaborations. The developers of the OASIS program are affiliated with the University of North Carolina (UNC) Center for Excellence in Community Mental Health, at the Department of Psychiatry at UNC's School of Medicine. Thus, both OASIS programs have a close relationship with UNC. OASIS also has a close relationship with Duke University. Other organizations and agencies with which OASIS has developed strong partnerships include the Local Management Entities-MCOs for the region, crisis intervention teams, law enforcement, community/family resource courts, and area health education centers.

Outreach to Potential Referral Sources. OASIS conducts regular outreach to the access units within the Local Management Entities-MCOs, as well as local emergency departments and hospitals, small medical practices, high schools, and colleges/universities (UNC and Duke University are an important source of referrals). OASIS is also working with Alliance, the Local Management Entity-MCO in Wake County, to educate high school counselors and improve early recognition of psychotic symptoms in high school settings. OASIS also promotes awareness about the program through periodic mailings and web-based trainings held for locally based clinicians (including pediatricians, family practice physicians, psychologists, family medicine), which they anticipate expanding to clinicians statewide.

Texas

As previously mentioned, the state has two newly developed CSC programs, one based at MHMRA of Harris County, a Houston-based LMHA, and the other based at Metrocare Pathways Clinic, a Dallas-based NTBHA provider. While both programs closely adhere to the CSC model, the Dallas Metrocare CSC program (Enhanced Program for Early Psychosis or ePEP) also provides cognitive remediation. The CSC program based at MHMRA of Harris County was able to become operational sooner than the Dallas Metrocare CSC program. This was partly a result of the MHMRA repurposing existing staff, which the Dallas Metrocare CSC program was unable to do because the agency was operating at capacity.

CSC Program at MHMRA of Harris County

The CSC program based at MHMRA of Harris County began recruiting patients in November 2014. At the time of our site visit, the program had two fully staffed teams with a total caseload of 61 clients.

Partnerships and Collaborations. The agency has a close partnership with the University of Texas (UT) Medical School at Houston. Researchers from UT's Department of Psychiatry and Behavioral Sciences are leading the subcontract to design and implement the evaluation of the newly developed program. The existing data sharing agreement between UT and the agency will greatly facilitate the evaluation; moreover, UT and its medical center are an important source of referrals for the program.

Outreach to Potential Referral Sources. In the initial phase of client recruitment, the CSC program restricted its efforts to reviewing lists of current LMHA clients to identify potential CSC clients based on age and diagnosis. The clinic also received some of its current clients through the Harris County Psychiatry Center (a local county hospital) and the Harris County eligibility center (central location for individuals in the community to come and request LMHA services). The clinic plans to take advantage of previously established co-located programs--LMHA clinical staff provide child services near schools--for recruitment. The deputy director for MHMRA of Harris County has advertised the services to all the other clinics in the area, and the CSC program has gotten referrals from those clinics as well.

Enhanced Program for Early Psychosis (Dallas CSC program)

The ePEP, based at the Metrocare Pathways Clinic in Dallas, began recruiting patients in February 2015. At the time of our in-depth telephone discussion, the program had one fully staffed team and another with most of the staff in place, and a total caseload of 36 clients.

Partnerships and Collaborations. The agency has a close partnership with UT and its Southwestern Medical Center located in Dallas, where the lead investigator for the CSC evaluation is a professor of psychiatry. ValueOptions, the carve-out managing the care provided by ePEP and all behavioral health providers in the NTBHA, is an important partner in the program implementation, as it provides the CSC with lists of potential clients in the program's catchment area and it also collects clinical and cost data that may be used for the evaluation component.

Outreach to Potential Referral Sources. Thus far, client recruitment efforts have included advertising CSC services internally (Metrocare Pathways Clinic, which has several sites) and at local hospitals (Timberlawn, Green Oaks, Terrell State). Program staff also regularly search through lists of potential clients provided by ValueOptions.

Virginia

As already mentioned, Virginia developed eight new CSC programs. Although the state indicated that CSC services could be offered to a broadly defined ESMI population ages 16-25 that included people with non-psychotic disorders, several programs were more restrictive in their clinical target population. However, they expressed flexibility with regard to age, accepting younger and older clients.

The expected start date for the CSC programs was October 1, 2014, and the programs were expected to be fully operational by June 30, 2015. Several programs were already operational before that date. Only one grantee was unable to meet that expectation because it had planned to do a sole-source contract with a community provider but the county had instructed the grantee to do a public solicitation of services.

Seven programs are using the OnTrackNY training curriculum, one CSC program (Highlands CSB) is exclusively using the NAVIGATE training curriculum because NAVIGATE seemed better aligned with the program's rural setting, and another program (Western Tidewater CSB) is using a combination of both curricula.

While the state did not have an expectation that the grantees would need to tap into additional funding sources to operate, several CSC programs were expecting to bill private insurance for billable services.

Alexandria Department of Community and Human Services (DCHS)/CSB CSC Program

In addition to all CSC components, the Transitioning Adults into Living Successfully (TRAILS) CSC program in Alexandria provides permanent supported housing to approximately ten clients per year through a contract with a private, non-profit community provider. The program is located in the West End Wellness Center, a psychosocial rehabilitation center located within the CSB's Center for Adult Services, but it will work in close collaboration with the CSB's Center for Children and Families.

The program's current target population is individuals with any early psychotic disorder within five years of symptom onset. However, program staff is considering stricter eligibility criteria to be better aligned with the CSC model--that is, include only FEP of onset within two years of intake. While the target caseload is 30 clients, at the time of the site visit, the CSC program had enrolled four clients and had 7-10 referrals.

Partnerships and Collaborations. The Alexandria CSB has a close relationship with a state hospital within its catchment area, the Northern Virginia Mental Health Institute. The agency also partners and collaborates with the local public schools; specifically, the school social workers--CSB staff regularly attends individualized educational program meetings and other school activities, and a CSB-affiliated mental health professional is among the staff at the TC Williams High School's ninth-grade Teen Wellness Center.^y Lastly, the CSB has a partnership with the court system and deploys mental health professionals in the court services unit.

Outreach to Potential Referral Sources. At the time of our site visit, most of the outreach efforts had targeted emergency services and the Northern Virginia Mental Health Institute. The CSB also has reached out to the local Family Assessment Planning Team^z to discuss potential outreach strategies. The agency also plans to seek referrals from the local public and private high schools and it will reach out to student health services at local community colleges and nearby universities (e.g., George Washington University, George Mason University, Howard University). The CSB plans to leverage its partnership with the local court services unit for referrals.

Fairfax-Falls Church CSB CSC Program

The Fairfax-Falls Church CSB implemented a CSC program in collaboration with the Fairfax Psychiatric Rehabilitation Services (PRS). The target population for the Fairfax-Falls Church CSC program is individuals diagnosed with any early psychotic disorders within two years of diagnosis. The program's rationale for selecting a relatively broad diagnostic category was that they did not want to exclude clients who might benefit from CSC services. However, the program will consider focusing on FEP and excluding affective psychotic disorders, given that the evidence base is scarcer for the latter group, if the caseload grows rapidly and they reach capacity too soon.

Although program staff had initially planned a target caseload of 80 clients within the first year, they decided to reduce that number to 60, as a result of knowledge gained during an OnTrackNY training. At the time of our site visit, the CSC program had enrolled seven clients and had 38 referrals.

Financing Strategy. The CSC program expects that it will be able to bill private insurance for billable services as a result of many young people being covered under their parents' insurance and the "level of affluence" in the county.

Partnerships and Collaborations. The Fairfax-Falls Church CSB is partnering with the Fairfax PRS, to develop its CSC program. The CSB and PRS have had a close partnership for about 50 years. The CSB has contracted with PRS to provide psychosocial day programs and vocational services for CSB clients.

Outreach to Potential Referral Sources. Program staff have reached out to other CSB programs, crisis services, the main hospital system in Fairfax, public high schools, the Northern Virginia Community College and two local universities (Marymount University and George Mason University), a local jail, and the Offender Aid and Restoration program,^aa among other social service agencies. Program staff is looking for listservs listing local pediatricians and is considering outreach strategies to target local CMHCs and federally qualified health centers. Finally, the CSC program advertises its services through its own website (http://turningpointcsc.org/).

Highlands CSB CSC Program

The target population for the Highlands CSB CSC program is individuals with FEP within two years of symptom onset. However, if the program does not grow too rapidly, the CSB will consider expanding the clinical eligibility criteria to serve clients in their second psychotic episode or with early bipolar disorder.

At the time of our in-depth telephone discussion, the CSC program had only one hired team member (the team leader); however, the program's expectation was that they would become operational by June 30, 2015.

Financing Strategy. The CSC program expects to bill third-party payers to ensure future sustainability of the program.

Partnerships and Collaborations. Since the Highlands CSB is located in a rural area with low population density, the CSB has cultivated strong partnerships and collaborations to build a robust referral network for client recruitment. The CSB deploys staff members in local schools, including day treatment programs in some of the high schools. The agency has had agreements for many years with a private college and a community college located within the agency's catchment area to address the behavioral health needs of their student bodies. For example, CSB staff regularly meets with the private university's counseling staff and is on hand for students in crisis. The CSB also partners with family advocacy groups.

Outreach to Potential Referral Sources. The CSB plans to leverage its partnerships and collaborations to recruit clients for its CSC program.

Western Tidewater CSB CSC Program

In addition to all CSC components, the Western Tidewater CSC program will include a 90-day independent training program, where clients experiencing difficulties with independent living will be provided with furnished apartments for 90 days as part of an intensive program to build skills for independent living.

Although the CSC program's primary target population is individuals with any early psychotic disorder, the program will also accept individuals without psychosis as long as they have ESMI.

At the time of our in-depth telephone discussion, the CSC program had enrolled six clients.

Partnerships and Collaborations. Western Tidewater CSB is located in a largely rural area and, as a result, has developed strong partnerships and collaborations to build a robust referral network for client recruitment. The CSB has partnerships with primary care providers and schools. Western Tidewater has long collaborated with court services units and law enforcement agencies; the agency employs two mental health court service liaisons and hosts a crisis intervention training program for law enforcement, dispatchers, etc., to teach them how to interact with mentally ill individuals.

Outreach to Potential Referral Sources. In addition to conducting internal outreach to look for referrals from within the CSB, the agency plans to leverage its partnerships and collaborations to recruit clients for its CSC program. The CSB recently reached out to the local Family Assessment Planning Team (see footnote on proximal referral sources in the section "Implementation Strategy as Designed by the ESMI Programs") to discuss possible referral sources.

Wisconsin

As already mentioned, the only grantee in the state is JMHC, a large and established provider located in Madison in Dane County. JMHC enrolled its first client on March 6, 2015; at the time of our second discussion with program staff, JMHC was fully staffed although it was still recruiting a peer support staff, and the program had enrolled eight clients.

Partnerships and Collaborations. JMHC has a close relationship with the local community hospitals and had close prior relationships with the research staff of University of Wisconsin, Madison, and was able to recruit a researcher and a graduate student to assist with the evaluation component of the program.

Outreach to Potential Referral Sources. In addition to conducting internal outreach to look for referrals from within JMHC, the agency plans to tap into its extensive referral network and leverage its partnerships and collaborations to recruit clients for its CSC program. The agency will seek referrals from discharge staff at inpatient units, emergency departments and the JMHC Emergency Services Unit, primary care practices, local child and youth mental health programs, three large public high schools and alternative high schools, the University of Wisconsin-Madison, local health care providers, including the University of Wisconsin-Madison Medical Center, local shelters, the criminal justice system, and churches.

Program Strategy: Tier 3 States

Colorado

The state awarded the set-aside contract to the Jefferson Center for Mental Health in June 2015, and by the time of our telephone discussion, the grantee had not yet begun to hire staff. The state anticipates that the CSC program will become operational by fall 2015.

Iowa

The state awarded the set-aside contracts to two CMHCs in April 2015. As of June 2015, the programs were hiring and training staff but neither team had enrolled clients. Although the trainings are provided jointly to both programs, they differ in some respects--for example, only one of them is planning to offer supported employment services.

Partnerships and Collaborations. One of the grantees has a close relationship with a local substance abuse provider organization--the CSC program is leveraging the partnership to ensure access to substance abuse services for its future clients.

Outreach to Potential Referral Sources. One of the CSC programs is exploring the use of social media tools as a recruitment strategy. Also, at the time of our telephone discussion, one of the programs was planning a presentation to a local community group that includes hospitals, faith-based organizations, and youth groups among others, to develop a referral base.

Nevada

The Children's Cabinet, a Reno-based provider, is the state's sole set-aside grantee and FEP program. At the time of our site visit, the program was not yet operational but extensive planning had been conducted over the previous year and hiring for staff was beginning. The program anticipates becoming operational later in 2015.

Partnerships and Collaborations. The Children's Cabinet has an extensive network of community partners, including local hospitals, a mobile crisis program for adolescents funded through the Department of Children and Family Services, primary care providers, the Washoe County School District and its health center, and family organizations. The program model integrates staff from the Children's Cabinet with state-employed clinicians.

Outreach to Potential Referral Sources. The Children's Cabinet plans to leverage its partnerships and collaborations to recruit clients for its FEP program. Particularly valuable outreach opportunities include suicide screenings for adolescents and meetings with primary care providers and families. The program will be able to reach a statewide audience at the Mental Health Summit held annually in Reno.

Washington

The Central Washington Comprehensive Mental Health, a large multispecialty human services provider, is the state's sole grantee; New Journeys, the ESMI program modeled after the EASA model and located in Yakima County, targets individuals with FEP. The program anticipates becoming fully operational by September 2015.

Partnerships and Collaborations. The agency has strong connections with multiple community partners, including law enforcement.

Outreach to Potential Referral Sources. During the initial phase of implementation, the program expects to seek referrals largely from within the agency, in particular from emergency departments, crisis intervention teams, and inpatient psychiatric services. Over time, depending on capacity, the program expects to seek referrals from a broad range of community partners.

Evaluation Strategy

Description of the States' Evaluation Strategy

To ensure that ESMI programs paid for by the set-aside funds implement interventions that yield the desired outcomes, the SAMHSA guidance to states includes language related to collecting data with the goal of "demonstrating program effectiveness."¹ SAMHSA did not provide directives about how to conduct the program evaluation, and based on reports from states, additional guidance on the agency's "expectations for data collection and reporting" promised as part of the guidance did not follow. The guidance did highlight policymakers' interest in clinical and social outcomes, such as prevention of deterioration of cognitive functioning and fewer disruptions of employment. Given the states' discretion in strategy for using the set-aside funds to evaluate the ESMI programs as implemented, we explored whether the states have an identifiable evaluation strategy--designed by the states, grantees, or both--and whether separate set-aside funds have been carved out for evaluation. We also explored whether programs are leveraging academic partnerships and existing state information systems (see Table 4).

Here, we describe the states' plans for evaluating fidelity and outcomes, whether intermediate and typically measured as part of assessments of quality of care, or person-level outcomes capturing health status and well-being (see Table 4). Fidelity is commonly described with structure domains, intermediate outcomes are described with process domains, and person-level outcomes are described with outcome domains. We explored the states' approach for selecting evaluation domains targeted for the evaluation, data sources for the construction of measures, and the frequency of assessments. Because Tier 1 and Tier 2 states were farther along in their program implementation, we describe evaluation plans by tier. Our exploration of domains for the program evaluation component was designed in collaboration with NIMH.

Program fidelity is the degree to which the program is implemented "as intended," according to the known characteristics of the program model. Our examination of states' plans for evaluating fidelity included whether the strategy captured the following structural domains: staffing, caseload size, team orientation, off-hours availability, and outreach. Similarly, our examination of states' plans for evaluating intermediate outcomes and quality of ESMI care included whether the strategy captured the following process domains: time from referral to intake; identifying and addressing suicide risk and substance use; quality of engagement; adequacy of family involvement, metabolic testing, and pharmacological management; and frequency of medication management visits. Last, we determined if states sought to capture the following person-level outcomes domains: incidence of suicidal behavior, symptom stability, substance abuse behavior, schooling/employment, and criminal justice system involvement.

TABLE 4. Evaluation Strategy by State
By Tier and State	State Prescriptive About Strategy	ESMI Programs Contacted	Fidelity Monitoring Plan (Fidelity Domains)	Outcomes Monitoring Plan (Process and Outcomes Domains)	Other Aspects of Evaluation Strategya	Data Sources
TIER 1
California	No	Alameda	Plan not well defined yet	Process Domains: Quality of service delivery Outcomes Domains: Housing status Employment status	Partners/Consultants: Actively engaged in development of evaluation strategy with program developers (The Felton Institute)	County record keeping on homelessness and employment, clinical records from FSP
		Fresno's FOT	Plan not well defined yet	Process Domains: Hospitalization Use of crisis services Outcomes Domains: Employment status	Not specified	EHRs Primary data collection, Medicaid
		Imperial	Plan not well defined yet	Outcomes Domains: Criminal justice involvement Employment status Housing status	Partners/Consultants: Using evaluation methods from PIER program. Hired consultant on evaluation.	EHR (AVATAR) Primary data collection
		San Luis Obispo	Plan not well defined yet	Outcomes Domains: Educational status	Partners/Consultants: Working with CalPoly to develop evaluation plan	Data source(s) not well specified
Connecticut	No	IOL	Fidelity Domains: Clinician skill Clinician comfort with model	IOL program is not monitoring process or outcomes of care	Not specified	Data source(s) not well specified
		STEP	Fidelity Domains: Clinician skill Clinician comfort	Process Domains: Engagement DUP Hospitalizations Outcomes Domains: Employment status Quality of life Cardiovascular health Social functioning and recovery Community tenure Severity of symptoms	Frequency: Process and outcomes domains are routinely assessed; exact frequency was not specified.	Data source(s) not well specified
New York	Yes	Northern Rivers Family Services	Fidelity Domains: Staff composition Off-hours availability Frequency of team meetings and communication	Process Domains: Adequacy of: Substance use care Family counseling Antipsychotic treatment Outcomes Domains: Symptom stability Educational status Employment status Quality of life	Frequency: Process and outcomes domains are routinely assessed; exact frequency was not specified. Partners/Consultants: Working with OnTrackNY experts at Center for Practice Innovations to develop evaluation methods.	EHR
TIER 2
Idaho	Yes	St. Luke's Regional Health Center	Plan is in development	Process Domains: Participant satisfaction Hospital readmissions Emergency department visits Outcomes Domains: Criminal justice system involvement Symptom severity Problem drinking	Standardized Scales: PHQ, AUDIT, and functioning scales Frequency: Data collected at baseline, routinely, and at discharge; functioning scales administered every 90 days	St. Luke's EHR
North Carolina	No	OASIS (including OASIS Carrboro and Wake STEP)	Fidelity Domains: Staff-to-patient ratios Quality and frequency of individualized treatment planning Quality and frequency of multidisciplinary team meetings	Process Domains: Quality of: Team collaboration Integrated substance abuse counseling Pharmacological care Outcomes Domains: Housing status Employment status Symptom severity	Standardized Scales: EP-TAP Brief Assessment of Cognitionb Frequency: Client-level data--baseline and every 6 months; Standardized assessments--quarterly; Suicide risk--routine assessment Partners/Consultants: Working with experts at Center for Excellence in Community Mental Health, Department of Psychiatry, UNC to develop evaluation methods.	NC Tracksc (Medicaid information system) TOPPSc NOMSc Alpha (internal data system) EPIC (hospital-based EHR)
Texas	Yes	ePEP (Dallas Metrocare)	Fidelity Domains: Adequacy of: Staffing Caseload size Team orientation Off-hours availability Outreach	Process Domains: Crisis history Psychiatric hospitalizations Clinic visits (any care) Appointments missed Mental health services Outcomes Domains: Residential status Criminal Justice involvement Employment status Symptom severity Risk behaviors Behavioral health needs Social functioning	Standardized Scales: CANS, ANSA, PANSS, MASQ-D30, and Birchwood SFS Frequency: CANS and ANSA administered at intake, every 90 days, and at discharge; other measures collected during baseline assessment, interval assessments every few months and final outcomes assessment at the end of the year. Partners/Consultants: Contracted with UT Southwestern Medical Center to conduct evaluation.	CMBHSc
		MHMRA of Harris County	Fidelity Domains: Adequacy of: Staffing Caseload size Team orientation Off-hours availability Outreach	Process Domains: Crisis history Psychiatric hospitalizations Outcomes Domains: Symptom severity Risk behaviors Behavioral health needs Social functioning	Standardized Scales: CANS, ANSA, BPRS, PANSS, Columbia suicide rating scale. Frequency: CANS and ANSA administered at intake, every 90 days and at discharge; BPRS and PANSS administered during program intake, after three months of services, and after 1 year of services. Partners/Consultants: Contracted with UT Medical School at Houston to conduct evaluation.	CMBHSc
Virginia	No	Alexandria CSB	Fidelity Domains: Supervisory notes Team meeting logs EHR will be used to track fidelity Team supervisor will monitor fidelity on a regular basis.	Process Domains: Hospitalizations Engagement Family engagement Outcomes Domains: Criminal justice involvement Employment status Education status Symptomatology Satisfaction with services Measure of recovery	Standardized Scales: As determined by the work group, all CSBs will use the modified Colorado index and a self-reporting tool that includes components of the Mental Health Consumer Satisfaction Youth Satisfaction Survey and the Illness Management and Recovery Self-Report Scale Frequency: Outcome measures from CSBs every quarter and update the information in the state-level data repository system. Partners/Consultants: Fairfax CSB contracted with an outside evaluator to develop evaluation plan. Highlands Program is working with the NAVIGATE team and has contracted with an outside consultant to develop evaluation methods.	NOMSc and TEDSc
		Fairfax-Falls Church CSB	Plan is in development with assistance from an outside evaluator
		Highlands Community Services	Fidelity Domains: Weekly team meetings Quarterly meetings with the advisory committee Technical and administrative assistance calls with the NAVIGATE team Input from external consulting company
		Western Tidewater CSB	Plan not well defined yet
Wisconsin	No	JMHC	Fidelity Domains: Outreach and engagement visits reviewed on a monthly or quarterly basis Program participants seen by CSC providers Supervision logs Individual/team trainings during monthly meetings	Process Domains: Hospitalizations Outcomes Domains: Employment status Criminal justice system involvement Housing status Suicidality Meeting self-set goals Substance use	Standardized Scales: Reach and Recovery scale, PANSS Frequency: Reach and Recovery and PANSS every 6 months; progress of clients reviewed weekly Partners/Consultants: Partnered with the University of Wisconsin, Madison, to develop evaluation plan.	JMHC's EHR (Omnis)d
TIER 3
Colorado	N/A	Jefferson Center for Mental Health	Plan not well defined yet	Plan not well defined yet	Not specified	Data source(s) not well specified
Iowa	N/A	CMHC, Des Moines	Plan not well defined yet	Plan not well defined yet	Not specified	Data source(s) not well specified
		CMHC, Cedar Rapids	Plan not well defined yet	Plan not well defined yet	Not specified	Data source(s) not well specified
Nevada	N/A	The Children's Cabinet of Northern Nevada	Plan not well defined yet	Plan not well defined yet	Partners/Consultants: Developing evaluation plan with Oregon (EASA) consultants.	Data source(s) not well specified
Washington	N/A	Central Washington Comprehensive Mental Health	Fidelity Domains: Measurement methods used in Oregon's EASA model	Process Domains: Hospitalizations Outcomes Domains: Employment status Educational status	Partners/Consultants: Working with researchers at the University of Washington to develop evaluation plan.	Data source(s) not well specified
Includes any standardized scales used during assessment; frequency of assessment (e.g., baseline, routine/multiple, at discharge); control groups, and any academic partners/consultants. EP-TAP was funded to provide technical and evaluative assistance to the 3 North Carolina sites. Key components of OASIS' EP-TAP include maintaining a quality assurance database, routine outcomes monitoring, and feedback-informed treatment protocols, such as the Brief Assessment of Cognition. Program is leveraging state information systems. Program is leveraging state information systems; collected data will be transferred to the state's centralized web-based data system monthly.

States' Evaluation Plans: Tier 1 States

California

Consistent with the highly decentralized nature of the state, the SMHA was not prescriptive about the evaluation component of the implementation of the set-aside policy. Each of the California counties with which we held discussions included an evaluation component, but they were at different development stages. Alameda and San Luis Obispo are working with academic partners to develop and implement evaluation methods (The Felton Institute and Cal Poly, respectively), and Fresno and Imperial are developing evaluation plans with input from consultants.

Fidelity. Although all counties have access to information on several relevant structural domains, they did not have well-defined plans for fidelity monitoring.

Process and Outcomes of Care. Alameda County plans to evaluate quality of delivery of clinical services, but respondents did not provide details. Fresno County plans to evaluate hospitalization and use of crisis services. No other county reported active efforts to assess process domains. All counties are planning to assess person-level outcomes. Alameda County plans to assess living situation and employment status at intake and annually until discharge--however, the county is considering more-frequent assessments. Alameda County uses clinical records from the FSP as a data source to evaluate outcomes. Fresno tracks employment status. Imperial County collects data on criminal justice involvement, employment, and housing status. Fresno and Imperial counties primarily use EHR data (AVATAR,^ab a new system in the case of Imperial County) supplemented with primary data collection and Medicaid data sources to evaluate outcomes. San Luis Obispo County routinely collects data on educational outcomes such as attrition rates and academic success.

Connecticut

The SMHA was not prescriptive about the evaluation component of the implementation of the set-aside policy in the state. While the IOL program did not have a well-defined evaluation strategy, the STEP Program has been involved in active evaluation projects designed to document the effectiveness of the model for many years. The IOL program does not have academic partners but the program collaborates with Hartford Hospital's research center. The STEP program has leveraged its longstanding partnership with Yale University as a collaborative program of the University's Department of Psychiatry.

Fidelity. Both programs collect data on clinician skill and comfort with the model as their main targets for monitoring program fidelity.

Process and Outcomes of Care. The IOL program does not monitor process or outcomes at this point. The STEP program considers the assessment of process of care to be central to the program's evaluation efforts. Respondents reported that they evaluate engagement with the program but did not provide details on domains, data sources, or frequency of assessments. However, they are interested in examining demographic disparities in these intermediate outcomes. The program routinely collects data on a variety of person-level outcomes domains, including severity of psychotic symptoms, DUP, hospitalizations, vocational outcomes, and quality of life. The program also tracks cardiovascular outcomes, social functioning and recovery, and community tenure (i.e., community dwelling, institutionalized, etc.).

New York

The SMHA was highly prescriptive in its evaluation strategy, which included an allocation of set-aside funds for the conduct of the evaluation component. Like other aspects of the policy implementation, the evaluation component has benefited from the close collaboration between the SMHA and Dr. Lisa Dixon's team at the Center for Practice Innovations. The newly developed OnTrackNY Program at the Northern Rivers site is working with Dixon's team to develop an evaluation plan.

Fidelity. OnTrackNY programs tap into routinely collected data to assess all the fidelity domains mentioned above. For example, OnTrackNY routinely assesses whether teams are appropriately staffed and have off-hours availability, and whether the staff regularly meets as a team.

Process and Outcomes of Care. OnTrackNY programs evaluate process of care through measures that capture all the process domains mentioned above. For example, OnTrackNY assesses whether substance use is identified and addressed, whether the team meets with family members, and the adequacy of antipsychotic regimens. OnTrackNY also assesses the outcomes domains mentioned above, including symptom stability and education/employment status, and also assesses quality of life.

States' Evaluation Plans: Tier 2 States

Idaho

The SMHA was moderately prescriptive in its evaluation strategy. As reported by respondents, the state instructed St. Luke's Regional Health Center, its sole grantee, to use a specified amount of the set-aside funds to conduct data analysis and produce outcomes reports. The state required that demographic and diagnostic data be collected at initiation of services and that selected outcomes information be collected upon discharge. The design of the evaluation component has been conducted through a collaborative iterative process between the SMHA and the grantee. St. Luke's does not have academic partners at this point. The grantee is not planning to leverage existing state information systems for the evaluation component.

Evaluation of Fidelity. The state is working with St. Luke's to develop a plan to monitor fidelity.

Process and Outcomes of Care. The state and the grantee are working on the development of a plan to evaluate process and outcomes. The outcomes evaluation initiative would leverage newly collected data (i.e., demographic, diagnostic, and selected person-level outcomes) and possibly data routinely collected by St. Luke's. Currently, St. Luke's collects information pertaining to symptom severity, problem drinking, hospital readmission rates, emergency department visits, and interactions with law enforcement. These outcomes are assessed with standardized scales (e.g., the Patient Health Questionnaire [PHQ]) and functioning scales; the latter are administered every 90 days. The state is also interested in evaluating participant satisfaction.

North Carolina

The SMHA allocated a portion of the set-aside funds to conduct the evaluation component, and the agency identified and recommended several data elements that applicants should consider in the evaluation design, but the agency was not prescriptive in its evaluation strategy. The original OASIS Program at the Carrboro in Orange County was given additional funding to develop a technical assistance and evaluation component for the three North Carolina sites, which they named Early Psychosis Technical Assistance Program (EP-TAP). Key components of OASIS' EP-TAP include maintaining a quality assurance database, routine outcomes monitoring, and feedback-informed treatment protocols. OASIS has had a long-term operational and academic partnership with UNC. OASIS' evaluation strategy does leverage state information systems as it utilizes a newly developed state Medicaid information system (NC Tracks) and an outcomes performance system (TOPPS).

Evaluation of Fidelity. The EP-TAP monitors program fidelity by assessing staff-to-patient ratios, quality and frequency of individualized treatment planning and multidisciplinary team meetings, and other domains.

Process and Outcomes of Care. Process and outcomes of care are evaluated through the EP-TAP. Process domains include weekly multidisciplinary team collaboration, integrated substance abuse counseling, routine assessment of suicide risk, and evidence-based pharmacological care. Outcomes domains include housing and employment status and symptom severity assessed through standardized scales such as the Brief Assessment of Cognition. These data are then entered into NC Tracks. Sources of data for the outcomes evaluation component are National Outcomes Measurement System (NOMS), North Carolina Treatment Outcome Program Performance System (TOPPS), Alpha (internal data system), and EPIC (hospital-based EHR). Client-level data are collected at baseline and every six months to inform program-level planning and fidelity scaling. Client-reported and clinician-reported data are used on a regular basis to guide day-to-day interventions; other data including results of standardized assessments are used quarterly to inform individualized treatment planning.

Texas

The SMHA was moderately prescriptive in its evaluation strategy, which included the allocation of a portion of the set-aside funds to conduct the evaluation component and the requirement that programs collect and report fidelity-relevant information described in the CSC curriculum. However, the state did not prescribe the evaluation strategy and each program was free to develop a suitable plan. Academic partners of each of Texas' two grantees, MHMRA of Harris County and ePEP in Dallas, designed the components for the evaluation plan. While MHMRA of Harris County contracted with the UT Medical School at Houston to conduct the program evaluation, the Dallas-based ePEP contracted with the UT Southwestern Medical Center to conduct the program evaluation. The evaluation teams in both MHMRA of Harris County and ePEP program in Dallas are planning to leverage information collected by the CMBHS, the state's electronic web-based data system.

Evaluation of Fidelity. While the MMHRA of Harris County did not have a well-specified strategy for assessing fidelity, the ePEP program had a clear strategy as documented in the Metrocare Intervention plan. The ePEP program's approach for assessing fidelity to CSC's structure of care captures all the fidelity domains mentioned above, including monitoring the adequacy of staffing and caseload. Fidelity data will be collected through client and clinician interviews.

Process and Outcomes of Care. Neither program discussed plans for assessing intermediate outcomes with process domains. All publicly funded providers in Texas perform the Child and Adolescent Needs and Strengths Assessment (CANS) and Adult Needs and Strengths Assessment (ANSA) at intake, every 90 days, and at discharge. These scales collect information on a number of outcomes domains (e.g., symptom domains, risk behaviors, social functioning, psychiatric hospitalizations).^ac Both evaluation teams are considering the use of these assessment tools in addition to other outcomes measures. Outcomes information for MHMRA of Harris County will be collected using the following scales: Brief Psychiatric Rating Scale (BPRS), Positive and Negative Symptom Scale (PANSS), and the Columbia suicide rating scale. The BPRS and PANSS will be administered during program intake, after three months of services, and after one year of services. The evaluation team will also collect data on a control group of patients who meet the eligibility criteria but are not participating in the CSC program and are receiving usual care at the clinics affiliated with MHMRA of Harris County. The evaluation team will train MHMRA staff to administer these scales and collect data to ensure sustainability of the evaluation component; the evaluation team will analyze collected data and report back to the state.

The annual baseline assessment for ePEP in Dallas will collect information related to "...residential status, mental health service utilization, arrests and incarcerations, and employment/disability status" during the past year of treatment. Clinical records will be reviewed to collect information on "...number of clinic visits for all levels and types of care, number of appointments missed, number of emergency care visits and hospitalizations." The research team plans to compare up to 60 clients enrolled in the ePEP program to 60 control subjects who meet the eligibility criteria but are not participating in the CSC program and are receiving usual care at the Metrocare clinics. Individuals in the ePEP group and the control group will receive a baseline assessment, interval assessments every few months, and a final outcomes assessment at the end of the year. Assessments to collect outcome measures will be conducted by a study coordinator. Outcomes scales include: the PANSS, Mood and Anxiety Symptoms Questionnaire (MASQ-D30), and the Birchwood Social Functioning Scale (SFS). Participants will also complete sections of the baseline assessment at the end of a year. Clinical records will also be reviewed.

Virginia

Although the request for proposals issued by the SMHA to selected grantees stated that the agency would require "quarterly data reporting and outcome...with required data elements to be determined once SAMHSA guidelines for federal data reporting expectations are released," the SMHA was not prescriptive in its evaluation strategy and no distinct funds were allocated specifically for program evaluation. However, the strategy used to assess program effectiveness is evolving through a collaborative process between a state work group^ad and the eight grantees. This initiative had just begun by the time of our site visit, a delay the state attributed to their expectation that they should wait until the promised federal guidance on the evaluation component was issued. At this time, none of Virginia's CSBs have partnered with an academic institution for program development, implementation, or evaluative purposes. The grantees are planning to leverage information collected by the state systems through EHRs, however, not all CSBs in the state use the same EHR.

Evaluation of Fidelity. Supervisory notes and team meeting logs will track fidelity to the recovery orientation and team approach for the Alexandria Program. The EHR will be used to monitor that all CSC service elements are delivered according to client needs and preferences. Adherence to the model will be monitored by the team supervisor on an ongoing basis. The Fairfax CSB has budgeted for an outside evaluator to assist CSC program staff in developing data collection tools and fidelity models to ensure adherence to the RAISE model. Highlands Program plans to ensure fidelity through weekly team meetings, quarterly meetings with the Advisory Committee, and regular technical and administrative assistance calls with the NAVIGATE team. Highlands Community Service contracted with an outside consulting company, TurnKey KB & Associates, to assist with monitoring program fidelity and to serve as a liaison between Highlands Community Service, the SMHA, SAMHSA, and FEP/CSC experts.

Process and Outcomes of Care. The state does not have a clear strategy for collecting process domains. The SMHA's evaluation work group discussed possible outcomes domains and data collection strategy with the Center for Practice and Innovation and the eight set-aside providers in the state. The work group eventually decided on the following domains: hospitalizations/engagement with crisis or emergency services, criminal justice involvement, symptomatology, family engagement, satisfaction with services (self-report), measure of recovery (self-report), and engagement in employment or school-related activities. The work group decided to use a modified Colorado symptom index to collect measures related to symptomology. The state will also utilize data collected for NOMS and Treatment Episode Data Set (TEDS) reporting, which include hospitalization, employment and school engagement, and emergency department visits. For the self-report measures, a member of the data and evaluation group at the SMHA developed an instrument that includes components of the Mental Health Consumer Satisfaction Youth Satisfaction Survey and the Illness Management and Recovery Self-Report scale. The SMHA will collect the selected outcome measures from the CSBs on a quarterly basis and update information in a state-level data repository system. Currently, the CSBs are modifying their EHR systems to be able to collect the selected outcomes. Although the state did not specifically plan to collect measures related to suicidality and substance use behaviors, respondents mentioned that CSBs would be able to collect such data easily.

Wisconsin

Wisconsin was not prescriptive in its evaluation strategy, and no distinct funds were allocated specifically for program evaluation. Program fidelity will be overseen by the SMHA's contract manager, but JMHC, the sole grantee, developed an evaluation plan overseen by the agency's Continuous Quality Improvement Committee. JMHC has leveraged a valuable partnership with the University of Wisconsin, Madison, to pursue the evaluation. As previously mentioned, the outcomes evaluation component will be designed and overseen by a researcher affiliated with the university free of charge. Moreover, university-affiliated researchers provided JMHC with the resources needed to train the CSC staff on the administration of structured clinical interviews during the intake process. Data collected by JMHC is uploaded to the state's centralized web-based data system on a monthly basis.

Evaluation of Fidelity. To measure adequate staffing and outreach, JMHC will monitor the number of outreach and engagement visits completed monthly or quarterly and the number of program participants seen by CSC providers. Monthly meetings will be held to monitor program fidelity through a review of supervision logs and a review of individual/team trainings.

Process and Outcomes of Care. JMHC does not have a well-specified plan to assess process of care. However, the grantee has a clear strategy for evaluating outcomes domains. JMHC will tap into its EHR system, Omnis, to collect outcomes information including employment, hospital days, jail days, housing status, suicidality, substance use, and whether people are meeting self-set goals. Data is uploaded to the SMHA's centralized web-based data system on a monthly basis. The clinic is building a new EHR with additional fields to be able to capture more outcome domains for the FEP program. Measures are collected during clients' one-on-one visits with clinical staff, and progress of clients is reviewed every week. The prescriber will also collect measures related to the clinical symptoms of psychosis using the Reach and Recovery scale and PANSS every six months.

States' Evaluation Plans: Tier 3 States

Colorado, Iowa, Nevada, Washington

None of the Tier 3 states with which we held discussions were prescriptive with their program evaluation strategies. Nevada was the only Tier 3 state that specifically dedicated set-aside funds to the conduct of a program evaluation. The Washington SMHA is working with researchers at the University of Washington to develop a research and evaluation plan for the New Journeys Program based in Yakima County. Tier 3 states were unable to report whether their ESMI programs would leverage existing state information systems, as many states were still in the very early stages of developing evaluation plans.

Only Washington reported having a tentative plan for evaluating fidelity (adopting fidelity measurement methods used in Oregon's EASA model) and outcomes of care (monitoring hospitalizations, employment, and educational attainment); none had plans for evaluating process of care.

Barriers and Facilitators to Implementation

Barriers

Population Density. States face the challenge of matching programs with the appropriate populations, particularly in non-urban areas. Some states, such as Nevada, have avoided this issue for the moment and have established their initial ESMI programs in urban areas. However, this leaves their non-urban areas uncovered. Some states, such has Oregon, have decided to take on the challenge of establishing an ESMI program in a non-urban region, which requires referral and provider networks over a much larger area. Over time, these programs will face challenges in providing services to clients who must travel long distances unless they can develop alternative methods for providing services remotely. Future programs are likely to benefit from lessons learned in these early attempts to deliver services to non-urban populations with FEP and other ESMI.

Narrow Eligibility Criteria. FEP programs such as CSC have, by design, more narrowly defined eligibility criteria than most social service programs. While the reasons for focusing on non-affective psychosis are clear, this does create challenges for CSC programs--only one of which is identifying a large enough group of clients to make the program cost-effective. Since many of the organizations that refer to CSC programs do not have the clinical expertise to determine eligibility and because they are often searching for resources for the young adults they serve, a large portion of referrals are likely to fail to meet criteria for the CSC program. In many cases, programs will not simply be able to turn these clients away, but will want to refer them to other appropriate programs in the community. In addition, refusing to accept clients referred from community agencies may lead to misunderstandings among agencies. New programs may benefit from lessons that more-established programs have learned regarding managing the intake and eligibility screening process.

Hiring Staff. Workforce shortages are common in public mental health service systems across the country. Shortages are particularly acute in rural areas, but they also affect urban areas where clinicians have options that pay higher salaries than public sector psychiatry. These shortages may affect CSC and similar programs more acutely because of their need for clinicians to provide specialized therapy. CSC often has the option of using existing staff members who have related skills and can devote a portion of their time to the program, or hiring new staff who will be specialized in the CSC program. In many cases, the strategy of leveraging existing staff will be a more successful strategy because of the difficulty in hiring new staff members. This scenario is best illustrated by the Texas experience in which the CSC program that hired new staff took far longer to become operational relative to the program that hired within the parent agency.

Evaluation Capability and Focus. Grantees may have very limited data collection and analysis capability, which will undermine efforts to systematically evaluate the effectiveness of ESMI programs. Moreover, although SAMHSA had indicated to states that guidance on evaluation would be forthcoming, states have not received much of it to date. As a consequence, states have developed evaluation plans that vary widely in their scope and breadth. The lack of guidance clearly contributed to delays in the implementation of ESMI programs in Virginia, and may also have been a barrier in other states. Although all the states in this study were developing evaluation plans for their ESMI programs, they all faced considerable challenges in this area.

Facilitators

Strong State Guidance. Grantees in several states reported that they very much appreciated and valued state guidance and logistical assistance facilitated by the SMHA in connection with implementation and evaluation activities.

Existing Programs for Transitional Age Youth. States took advantage of existing programs serving at-risk TAY, some of which have connections to mental health treatment facilities, as well as a broad range of social support services. Although the TAY programs have a much broader target population than CSC and other ESMI programs, linking an ESMI program to a TAY program can facilitate implementation. The TAY programs may come into contact with young adults experiencing a FEP through their interaction with youth serving clinical and social services agencies. Second, TAY programs may include staff with skills that are valuable to ESMI programs, including expertise in care coordination. Several CSC programs in our case study states either help to cover the salaries of staff associated with existing TAY programs and/or provide specialized training in the identification of FEP to TAY program staff.

Existing Provider Networks. States have found it useful to draw on existing community-based networks of clinical and social service providers in establishing their ESMI programs. Most commonly, ESMI programs are situated within large social service agencies, which not only have a history of serving children or youth, but also have extensive connections to specialty mental health providers, schools, and other potential sources of services and referrals within their region. States may also take advantage of organized networks or coalitions of provider organizations. These networks can provide referrals to ESMI programs because they are likely to come into contact with young adults experiencing a first episode of psychosis or serious non-psychotic illness and have established lines of communication.

Existing Centers of Expertise in CSC and Other ESMI Models. Several states that had no previous experience with early intervention programs relied heavily on advice from consultants from established centers with expertise (e.g., Center for Practice Innovations at the New York State Psychiatric Institute, the Felton Institute in San Francisco). These centers provided input on a wide range of issues including program location, staff composition, referral networks, clinical skills, and evaluation methods.

Lessons Learned Concerning Evaluation

Several themes emerged from the case studies that might inform evaluation approaches for CSC and other ESMI programs being implemented across the country.

Model Fidelity. Most of the states started with a particular model of ESMI program, such as CSC, PIER or PREP, but the extent to which the models differ in meaningful ways and the extent to which each model is implemented with fidelity remains unclear. We expect that variation across programs, even those implementing the same model, will increase over time as programs adjust to the constraints of their local environments. Nonetheless, the programs still share a family resemblance as CSC programs. A characterization of the CSC or other model will be needed for an evaluation, but we recommend that this characterization allow for the natural variation across programs that can be expected to arise over time, rather than on a strictly defined set of model features. The model can be defined with sufficient detail to draw meaningful inferences about its effectiveness based on a multisite evaluation, but still allow for variations across sites that are needed to fulfill those functions in different settings. In fact, evaluations of specific model adaptations may contribute critical information to a nascent evidence base on the feasibility and effectiveness of community-based ESMI programs in the United States. In resource-richer environments, formative evaluations may provide valuable insights into the drivers of any such adaptations.

Process Evaluation Domains and Measures. Many of the activities of the ESMI teams that should be tracked in a process evaluation are new to the agencies hosting the model and may present challenges for reliable measurement. The broadly encompassing and individualized nature of the program will add further challenges because there is not a single set of providers or services that should be provided to all clients. Moreover, many of the services depend on collaborations with external providers, such as schools or job placements. Tracking activities of widely dispersed ESMI team members and external partners will be a daunting challenge that should be carefully considered in the design of the evaluation. Measures that focus on functions expected from CSC programs rather than specific activities are likely to be important. In addition, methods for data collection on processes that describe interactions between clients and various provider agencies will need to be carefully considered for each program.

Outcomes Evaluation Domains and Measures. As with any evaluation that includes outcomes measures, evaluation designers and policymakers should be careful in their selection of outcomes domains and exercise caution in drawing inferences about the effects of the intervention on client-level outcomes, given that the latter may be influenced by factors that are not under the providers' control. That said, a focus on critical short-term outcomes--including suicidal behavior, symptom stability, substance abuse behavior, and schooling/employment--is important as these are directly affected by components of CSC and other ESMI models. Most of the programs we visited had the capabilities of collecting primary data on these outcomes and/or tapping into data collected by state agencies and other entities for other purposes.

Monitoring the Referral Process. In all of the programs we visited, the referral process was in flux to some degree. This was not a mark of a failure but a predictable part of the program evolution over time. The newer programs had relatively narrow referral networks, in part because they want to control the rate of growth and in part because they do not want to invest in new referral sources while they are still focusing on providing a new set of services to their first clients. While the strength of these referral networks would be a likely target for evaluation, the stage of development of the program should be taken into account. In addition, if the evaluation will have a population focus (e.g., an attempt to measure the impact of a CSC program on the course of psychotic disorders and development of disability), then some method for assessing the referral process relative to the total population of new onsets of psychotic disorder in the program's catchment area will be needed. For instance, new cases of psychotic disorder could be identified in Medicaid claims data or in hospital discharge data and compared with referrals to the CSC program.

Control Group. Another likely challenge for an evaluation of CSC and other ESMI programs aimed at demonstrating effectiveness will be identification of a control group that can provide a valid comparison to assess the program's impact. With some exceptions, evaluation plans have not included a control group, which requires a higher degree of sophistication in the resources available for the evaluation and pre-implementation planning. However, inclusion of a control group should be considered for future evaluations to separate general trends over time from effects of the ESMI program.

Conclusion

All case study states embraced the set-aside policy as a mechanism for developing or expanding services for people with FEP and other ESMI. However, the case studies revealed wide variation among states in how the funds were used. Some states made concerted efforts to establish specific models of FEP programs--developing highly detailed plans and drawing extensively on consultation with outside experts. In other states--notably California, where the funds were divided into relatively small amounts and distributed among many county mental health departments--the funds were used to supplement existing programs serving the target population by providing training in FEP treatment or establishing new referral mechanisms. State-specific factors, such as the extreme decentralization of the administration of MHBG funds in California or the experience with the RAISE program in New York, appear to have had large impacts on decisionmaking across the states. The availability of other public funds, including state general revenue funds (such as the fortuitous infusion of a large amount of state funds into FEP programming in Virginia), also had a large impact.

Most states implemented CSC programs or modified existing programs to become CSC-like, and while many expanded their clinical population focus to disorders beyond FEP, states' main rationale for doing so was to broaden access and enhance sustainability. States also varied in the sophistication and state of implementation of their evaluation plans, but all of them were committed to the goal of assessing program fidelity and outcomes. Although most of the states had not yet developed sustainable models for their ESMI programs and were reliant on continuing MHBG support, all states--even Texas, where only uninsured individuals are currently eligible for services--expressed interest in eventually tapping into third-party reimbursement to cover at least some operational costs. In one case, questions about the reliability of continued support for the ESMI program led to very cautious decisionmaking regarding the use of set-aside funds.

At this early stage, it is possible to conclude that the set-aside policy is improving access to services for individuals experiencing their first episodes of SMI, particularly those experiencing non-affective psychosis. Across the country, states are developing or expanding ESMI services by drawing on existing program models and adapting them to meet local needs. The programs are being designed with an outlook that emphasizes population impact and they include, from their early planning stages, strategies and resources for program evaluation. As more programs become operational over the coming months, a client population whose complex health care needs had been largely unmet is likely to see a large expansion in the availability of services geared to address their needs. Although most states have implemented or are planning to implement programs that closely adhere to CSC or variants, and similarly, all are committed to evaluating their set-aside funded programs, there is a need for ongoing federal guidance on best practices for program implementation and evaluation to ensure success of the policy. Moreover, a declaration of federal commitment to the policy may assuage states' concerns about the sustainability of the policy, and evidence of success could lead to broader dissemination of ESMI programs supported by state or even private funds.

References

1. Substance Abuse and Mental Health Services Administration. Guidance for the Revision of the FY2014-2015 MHBG Behavioral Health Assessment and Plan. Washington, DC: U.S. Department of Health and Human Services; April 29. 2014. Retrieved September 25, 2015: http://www.samhsa.gov/sites/default/files/private/mhbg-5-percent-set-aside-guidance.pdf.

2. Goldman HH. Will health insurance reform in the United States help people with schizophrenia? Schizophr. Bull. 2010;36(5):893-894.

3. Lieberman JA, Dixon LB, Goldman HH. Early detection and intervention in schizophrenia: a new therapeutic model. JAMA. 2013;310(7):689-690.

4. Heinssen RK, Goldman AB, Azrin ST. Evidence-Based Treatments for First Episode Psychosis: Components of Coordinated Specialty Care. Rockville, MD: National Institute of Mental Health; 2014.

5. Addington J. The promise of early intervention. Early Interv Psychiatry. 2007;1(4):294-307.

6. Mihalopoulos C, Harris M, Henry L, Harrigan S, McGorry P. Is early intervention in psychosis cost-effective over the long term? Schizophr. Bull. 2009;35(5):909-918.

7. Dixon L. Assertive community treatment: Twenty-five years of gold. Psychiatr. Serv. 2000;51(6):759-765.

8. Goldman HH, Karakus, M, Frey, W, Beronio, K. Financing first episode psychosis services in the United States. Psychiatr. Serv. 2013;64(6):506-508.

9. Crumlish N, Whitty P, Clarke M, et al. Beyond the critical period: Longitudinal study of   8-year outcome in first-episode non-affective psychosis. Br. J. Psychiatry. 2009;194(1):18-24.

10. Harris MG, Henry LP, Harrigan SM, et al. The relationship between duration of untreated psychosis and outcome: An eight-year prospective study. Schizophr. Res. 2005;79(1):85-93.

11. Marshall M, Lewis S, Lockwood A, Drake R, Jones P, Croudace T. Association between duration of untreated psychosis and outcome in cohorts of first-episode patients: A systematic review. Arch. Gen. Psychiatry. 2005;62(9):975-983.

12. Malla AK, Norman RM, Joober R. First-episode psychosis, early intervention, and outcome: What have we learned? Can. J. Psychiatry. 2005;50(14):881-891.

13. Petersen L, Jeppesen P, Thorup A, et al. A randomised multicentre trial of integrated versus standard treatment for patients with a first episode of psychotic illness. BMJ. 2005;331(7517):602.

14. Fowler D, Hodgekins J, Howells L, et al. Can targeted early intervention improve functional recovery in psychosis? A historical control evaluation of the effectiveness of different models of early intervention service provision in Norfolk 1998-2007. Early Interv Psychiatry. 2009;3(4):282-288.

15. Mueser KT, Penn DL, Addington J, et al. The NAVIGATE program for first-episode psychosis: Rationale, overview, and description of psychosocial components. Psychiatr. Serv. 2015;66(7):680-690.

16. Dixon LB, Goldman HH, Bennett ME, et al. Implementing coordinated specialty care for early psychosis: The RAISE connection program. Psychiatr. Serv. 2015;66(7):691-698.

17. Essock SM, Goldman HH, Hogan MF, Hepburn BM, Sederer LI, Dixon LB. State partnerships for first-episode psychosis services. Psychiatr. Serv. 2015;66(7):671-673.

18. Viebeck E. U.S. mental health chief says treatment cuts the risk of violence. The Hill. 2013.

19. Attias M. Senators Discuss Funding, Other Issues in Mental-Health Hearing. CQ News. January 24, 2013.

20. Sebelius K. Bring mental illness out of the shadows. USA Today. February 4, 2013.

21. Huskamp HA. The Affordable Care Act's dependent coverage provision and young adults with psychiatric disorders. Am. J. Psychiatry. 2015;172(2):113-114.

22. Karakus M, Frey W, Goldman H, Fields S, Drake R. Federal Financing of Supported Employment and Customized Employment for People with Mental Illnesses: Final Report. Rockville, MD: Westat, Inc.; 2011.

23. Goldman H, Karakus M. Do not turn out the lights on the public mental health system when the ACA is fully implemented. J. Behav. Health Serv. Res. 2014;41(4):429-433.

24. De Maio M, Graham P, Vaughan D, Haber L, Madonick S. Review of international early psychosis programmes and a model to overcome unique challenges to the treatment of early psychosis in the United States. Early Interv Psychiatry. 2015;9(1):1-11.

25. Dresser K, Clark HB, Deschenes N. Implementation of a positive development, evidence-supported practice for emerging adults with serious mental health conditions: The transition to independence process (TIP) model. J. Behav. Health Serv. Res. 2015;42(2):223-237.

Appendix

Identification of Grantees

TABLE A1. Identification of Grantees
State	Number of Grantees	Grantee Description
California	Exact number is not known because implementation is at the county level. 4 interviewed	San Luis Obispo FEP Program San Luis Obispo County Affiliated with: Cal Poly Fresno First Onset Psychosis Program Fresno County Affiliated with Felton Institute of San Francisco Alameda PREP Program Alameda County Imperial FSP Program Imperial County Imperial's Youth and Young Adult Program
Colorado	1 total 1 interviewed	Jefferson Center for Mental Health Jefferson County
Connecticut	2 total 2 interviewed	STEP Program New Haven County Affiliated with Yale University IOL Program Hartford County
Idaho	1 total 1 interviewed	St. Luke's Regional Health Center Twin Falls County St. Luke's Magic Valley Medical Center
Iowa	2 total 2 interviewed	CMHC Des Moines Iowa Department of Human Services CMHC Cedar Rapids Iowa Department of Human Services
Nevada	1 total 1 interviewed	The Children's Cabinet of Northern Nevada Reno Washoe County
New York	~30 total 1 interviewed	Northern Rivers Family Services Albany County
North Carolina	3 total 2 interviewed	OASIS, Carrboro, Orange County Clinic established by the UNC Center for Excellence in Community Mental Health, within the Department of Psychiatry at UNC-Chapel Hill's School of Medicine Wake STEP (extension of OASIS) Clinic Raleigh-Durham Wake County
Texas	2 total 2 interviewed	MHMRA of Harris County Harris County CSC program located in Southeast Clinic, LMHA Dallas Metrocare Dallas County CSC program (ePEP) located in the Metrocare Pathways Clinic, NTBHA
Virginia	8 total 4 interviewed	Alexandria DCHS/CSB City of Alexandria CSC program (TRAILS) located in Center for Adults Fairfax-Falls Church CSB Fairfax County and cities of Falls Church and Fairfax CSC program: Turning Point Highlands Community Services Washington County and Bristol, Virginia Western Tidewater CSB Cities of Suffolk and Franklin and the Counties of Isle of Wight and Southampton
Washington	1 total 1 interviewed	Central Washington Comprehensive Mental Health Yakima County
Wisconsin	1 total 1 interviewed	JMHC Dane County

State-Specific Case Study Methods

Tier 1 States

California

An initial phone discussion was conducted with officials within the California Department of Health Care Services (DHCS), the agency responsible for administering the state's MHBG. That discussion highlighted the decentralized nature of decisionmaking regarding the use of MHBG funds for the set-aside, with most critical decisions made at the county level rather than at the state level. State-level officials were not able to provide detailed information on the policymaking processes occurring at the county level and did not yet have the county plans for using the set-aside funds, which counties will be submitting to the state over the summer and fall of 2015. To better understand use of the set-aside funding at the county level, we conducted additional discussions, by phone and in person during a site visit to Sacramento, with representatives of the California County Behavioral Health Directors Association.^ae We also worked with respondents at DHCS to select several counties for site visits. DHCS provided RAND with sections of seven county mental health plans that described the intended use of the set-aside funds. These counties were selected because they had submitted their plans to DHCS earlier than other counties. Of those seven counties, we arranged to have phone discussions with county-level policymakers from four: San Luis Obispo, Fresno, Alameda, and Imperial. These counties were selected because their plans suggested that they intended to use their set-aside funds to support new programs for ESMI. Thus, the counties described in this report are not representative of California counties as a whole; rather, they represent a group that initiated a CSC program using their set-aside funds. We were not able to systematically investigate uses of the set-aside funds across all of California's county behavioral health departments.

Connecticut

An initial phone discussion was held with officials within the Connecticut Department of Mental Health and Addiction Services (DMHAS), the agency responsible for administering the state's MHBG, on February 5, 2015. Follow-up discussions were then held during a site visit to the state on February 25, 2015. RAND's site visit to the Connecticut DMHAS provided background information pertaining to the state's two existing ESMI programs (the STEP Program and the IOL Program) and Connecticut's strategy for use of the set-aside funds under the new policy. Telephone discussions were held with each of the providers in the state on April 21, 2015. Respondents from the STEP Program included the program director; respondents from the IOL Program included the program director, manager, and outreach clinician and coordinator.

New York

The set-aside program was initially discussed with officials from the New York State Office of Mental Health, the agency responsible for administering the state's MHBG, and with a staff member from the state's CSC program, OnTrackNY, located in the Center for Practice Innovations at the New York State Psychiatric Institute. Following the initial call, a site visit was made to one of the newly funded OnTrackNY programs. The program we visited, located in a large human services agency in Albany, was still in early planning stages and had not yet fully staffed their program or begun to accept clients.

Tier 2 States

Idaho

One discussion was conducted by telephone while a second was conducted during a site visit. Respondents included members of the Idaho Division of Behavioral Health Services within the Idaho Department of Health and Welfare--the agency responsible for administering the state's MHBG--and staff from Idaho's funded provider, St. Luke's Regional Health Center, including the senior director in specialty medical practice, the clinical supervisor and team lead, and the regional psychologist. The initial telephone call with Idaho state officials was held in February 2015, and the in-person discussions with state officials and St. Luke's staff were held on April 14, 2015.

North Carolina

An initial discussion conducted by telephone was followed by in-person discussions with members of the North Carolina Department of Health and Human Services--the agency responsible for administering the state's MHBG--including the Division of Mental Health and Substance Abuse Services block grant manager and the program manager for the Quality Management Team, as well as staff from the OASIS Program, including the medical director of OASIS, the clinical instructor/program director of the OASIS Wake site, program director of the OASIS Carrboro site, a child and adolescent trained psychiatrist, a licensed clinical social worker and clinical therapist, and a data coordinator. The initial telephone call with state officials and the in-person discussions with state officials and OASIS staff were held in June 2015.

Texas

Three phone calls were conducted between February 2015 and March 2015. The site visit was conducted in April 2015. Discussions were held with four representatives from the Mental Health and Substance Abuse Division of the Department of State Health Services (the agency responsible for administering the state's MHBG), three ESMI program staff members from the ePEP based at the Metrocare Pathways clinic, two members of the program evaluation team from the UT Medical School at Houston Department of Psychiatry and Behavioral Sciences, three ESMI program staff members from the MHMRA of Harris County Southeast clinic, and one member of the program evaluation team from the UT Southwestern Medical Center at Dallas.

Virginia

Four phone calls were conducted between February 2015 and May 2015. The site visit was conducted in June 2015. Discussions were held with three representatives from the Office of Mental Health within the Department of Behavioral Health and Developmental Services (the agency responsible for administering the state's MHBG), three ESMI program staff members from the Alexandria CSB, three ESMI program staff members from the Fairfax CSB, one ESMI program staff member from the Highlands Community Services, and two ESMI program staff members from the Western Tidewater CSB.

Wisconsin

Two phone calls were conducted between January 2015 and June 2015. The site visit was conducted in April 2015. Discussions were held with four representatives from the Division of Mental Health and Substance Abuse Services within the Department of Health Services (the agency responsible for administering the state's MHBG) and four ESMI program staff members from the JMHC.

Tier 3 States

Colorado

The Colorado set-aside plan was discussed during a telephone call in February 2015 with staff from the Office of Behavioral Health within the Colorado Department of Human Services, the agency responsible for administering the state's MHBG. From that conversation, RAND learned that the state had been delayed in contracting with a CSC provider and was still preparing a request for proposals from potential provider organizations. Due to the very early stage of development of the program, a site visit was not conducted. A second telephone conversation was held with the state officials in June 2015 to collect information on subsequent steps towards establishing a CSC program.

Iowa

Two discussions were conducted by phone and one discussion was conducted during a site visit. While Iowa had two contracts in place with CMHCs to implement FEP (non-affective) programs at the time of the second phone call, both CMHCs were undergoing training and had not yet begun serving individuals. For this reason, we held a discussion with the state-level FEP program manager and MHBG planner. Respondents included members of the Division of Mental Health and Disability Services within the Iowa Department of Human Services (the agency responsible for administering the state's MHBG), including the MHBG planner and Community Services and Planning Bureau chief, and the state-level project manager for the set-aside grantees. The initial telephone call to discuss strategy for use of the set-aside funding was held in February 2015. The in-person RAND discussion was held in March 2015, and the telephone call with Iowa staff on behalf of contractors administering the two CMHC-based ESMI programs was held in June 2015.

Nevada

Information on the use of the set-aside funds in Nevada was collected during a phone discussion with officials from the state Division of Public and Behavioral Health (the agency responsible for administering Nevada's MHBG) and an in-person discussion with officials at the Children's Cabinet, an agency selected to implement an FEP program using the set-aside funds in Reno. Although the program at the Children's Cabinet was still in the planning phase and not yet operational, a site visit with a one-hour key informant discussion was conducted at the agency during a two-day site visit to the state focused on the MHBG.

Washington

Two telephone discussions were conducted with representatives from the Division of Behavioral Health and Recovery within the Washington State Department of Social and Health Services, the agency responsible for administering the MHBG. The first discussion was conducted in early March 2015 and included planning officials from the state department of mental health. At that point in time, the state had decided on an overall strategy for use of the MHBG set-aside funds, the provider organization had been selected and the state was consulting with experts in neighboring Oregon regarding design of the program. The second discussion was conducted in early June and, in addition to the state planner, also included the administrator from the provider agency that will be administering the program and a member of the consulting team from Oregon. At the time of the second discussion, many additional details regarding the implementation plan had been resolved but the team was not yet operational.

Notes

1. This term was not operationally defined by Congress or SAMHSA. We use the term ESMI to refer to three exclusive categories: FEP, a category that includes a number of non-organic and non-affective psychotic disorders (schizophrenia, schizoaffective disorder, schizophreniform disorder, delusional disorder, and psychosis not otherwise specified); early stages of affective psychotic disorders (bipolar disorder, major depressive disorder with psychosis); and early stages of any non-psychotic SMI with a gradual onset (e.g., obsessive-compulsive disorder).

2. Although the white paper was written by NIMH, SAMHSA had the opportunity to review it iteratively. The white paper was the result of a congressional mandate instructing SAMHSA and NIMH to work together to develop guidance for states regarding evidence-based treatment approaches.

3. NIMH contributed funds for technical assistance activities through an interagency agreement with SAMHSA. NIMH also funded a series of webinars and supported in-person training by CSC experts in five states.

4. The health home program is an optional Affordable Care Act initiative launched in January 2011 that integrates physical and behavioral health and long-term services and supports for Medicaid-eligible individuals with chronic conditions.

5. We were able to review sections of mental health plans describing the intended use of the set-aside funds for seven counties that had submitted their plans to the SMHA earlier than other counties. We selected the four-county sample from that larger set based on our assessment that they intended to use the funds to support new ESMI programs.

6. The state contracts with eight Local Management Entities-MCOs located in eight multicounty geographic areas to manage public funds. These MCOs have two contracts with the state, one for Medicaid and one for state (general revenue) and MHBG funds. MCOs contract with providers across the state for mental health and substance abuse services and intellectual developmental disability services.

7. Publicly funded community-based mental health services in Texas are provided through LMHAs, government entities serving low-income populations throughout the state. The exception to this publicly operated system is the Northeast region of the state, the NTBHA, where mental health services are delivered by a private sector-operated system managed by ValueOptions, a behavioral health carve-out.

8. Initially, the state also required that clients have a maximum of one year of treatment but that criterion was dropped before our case study.

9. Publicly funded community-based mental health services in Virginia are provided through 40 CSBs. While many CSBs are local government entities operating under a city or county charter, some cover multiple jurisdictional areas and are quasigovernmental entities. CSBs are considered non-profits, and their boards consist of representatives of all the counties and cities they serve. CSBs provide services either directly or through subcontracts with private providers.

10. Virginia Request for Proposals document, pg. 1.

11. Wisconsin's Request for Proposal document, pg. 5.

12. Any of two psychosocial rehabilitation programs with established funding streams (larger MHBG, Medicaid).

13. There are 17 CMHCs in the state that operate on a capitated basis through a Medicaid carve-out.

14. The state's SMHA contracts with 11 Regional Support Networks (RSNs) comprising one or more counties to provide public mental health services. The state's public behavioral health system is undergoing structural reorganization and the SMHA will directly contract with regionally operated carve-outs for public mental health and substance abuse services, instead of RSNs, beginning in 2016.

15. A small portion of the set-aside funds are being used to incorporate an FEP component into a suicide prevention program, and to purchase access to an online library of recovery-related materials for providers across the state.

16. The state is a pioneer in this regard: FEP programs were introduced in 2001 and currently cover more than 90 percent of the state's population.

17. The relatively narrow clinical criteria for the program raise an issue regarding screening of referrals and disposition for referred individuals who do not meet criteria. In Oregon, where the same inclusion criteria are in effect, one-half to two-thirds of referrals are screened out of the program although they may have serious mental health problems. The New Journeys program will be well positioned to refer these individuals to other services that are available locally because of the broad range of evidence-based practices offered by the parent agency. Of note, the legislature approved additional state funds to help with referrals for these individuals.

18. These include proximal referral sources (clinic/provider the program is nested within; community-based health care partners; local inpatient units and emergency departments) and more-distal ones (schools, colleges, and universities; primary care practices; social services agencies; law enforcement and jails; advocacy groups; and the community at large.

19. The Mental Health Services Act provides funds for mental health and substance abuse services through a dedicated state income tax.

20. The TIP model "prepares youth and young adults with emotional behavioral disturbances for their movement into adult roles through an individualized process, engaging them in their own futures planning process, as well as providing developmentally appropriate and appealing supports and services." (http://www.tipstars.org/Home.aspx)

21. Imperial County webpage: http://www.co.imperial.ca.us/behavioralhealth/index.asp?fileinc=crimhsa.

22. FSPs, a core component of the Mental Health Services Act, have a low staff to client ratio, full-time crisis availability, and a team approach that is a partnership between mental health staff and clients.

23. The PIER model comprises three phases: outreach and engagement; treatment recommendation, which may involve low dose medication; and multifamily groups that meet every two weeks. Case management services follow the participant for 18 months to two years.

24. Based on prior patterns of care at the university health center, the team expects to serve 4-7 individuals a week and about 200 over the course of a year. The university administration also expects encounters in connection to students needing to be physically removed from campus ("transports").

25. Alexandria has 15 elementary schools, two middle schools, and one high school.

26. Each county in Virginia has a Family Assessment Planning Team, a partnership between the school system, CSB, and local court services that develops "recommendations for the treatment of children that are experiencing anything related to behavioral health, to academic, to criminal justice involvement, risk factor involvement, or are already an indicated treatment population."

27. The Offender Aid and Restoration program is a private, non-profit program that seeks to link ex-offenders coming out of jails to services.

28. See http://www.softwareadvice.com/medical/netsmart-avatar-suite-profile/ for more information on AVATAR.

29. SOURCE: Texas ANSA Manual--revised November 23, 2013.

30. The work group consisted of representatives from the Office of Mental Health, Office of Support Services, and the Office of Child and Family Services. The Office of Support Services has a data and evaluation team as they are responsible for managing performance contracts.

31. This organization advocates on behalf of county behavioral health departments at the state level on issues related to mental health and substance use treatment systems.

Project and Report Information

This report was prepared under interagency agreement #AMH14001 between the U.S. Department of Health and Human Services (HHS), Office of Disability, Aging and Long-Term Care Policy (DALTCP) and the National Institute of Mental Health. For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/office-disability-aging-and-long-term-care-policy-daltcp or contact the ASPE Project Officer, Kristina West, at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. Her e-mail addresses is: Kristina.West@hhs.gov.