State-Specific Characteristics and Developments That May Influence Implementation of the Set-Aside Policy in the State
We took into account whether the states operated ESMI programs or had been home to RAISE study sites prior to the implementation of the policy, as well as their Medicaid expansion and 1915(i) adoption status (relevant because of the financing opportunities these Medicaid innovations bring with them). In addition, we identified state-specific characteristics and developments that have affected implementation of the policy or have the potential to affect its success in the 12 case study states (see Table 1).
California has an extremely decentralized system for distributing its MHBG funds, and the set-aside funds have been channeled into that system. This has two important implications for the use of the set-aside funds. First, decisions about programming are not made at the state level. Rather, the funds, along with much larger amounts of state funds from multiple sources, are directed to 57 county departments of mental health. The counties submit plans for use of mental health resources to the state for approval, but they have broad decisionmaking powers. Second, the set-aside funds are divided among all the counties that request them. While the total number of counties requesting funds was not available, the expectation was that the vast majority would. Because the set-aside funds were divided into relatively small amounts for each county, creating an entirely new program using those funds was not financially feasible for most counties.
Although a Denver-based clinic participated in the RAISE study, the ESMI program had been discontinued by the time the set-aside policy was implemented. Moreover, implementation of the policy was delayed due to bureaucratic issues and staff changes and, as a result, the request for proposals was only released in the summer 2015.
The state is the site of two ESMI programs, the Specialized Treatment Early in Psychosis (STEP) program and the Institute of Living (IOL) program, which predate the RAISE study and the set-aside policy (see the section on Strategy for Implementing the Set-Aside Policy).
Idaho has one of the lowest rates of mental health spending in the nation, which compounds the challenges of providing mental health services in a state with a large rural population spread over a vast geographic region.
Iowa has an approved Medicaid state plan amendment to implement two health home programs;d the state will use other MHBG funds to fund health home services for adults not eligible for Medicaid.
Nevada covers a very large geographic area, but more than 90 percent of its population is concentrated in two urban counties, Washoe County, which includes the city of Reno, and Clark County, which includes the city of Las Vegas. The state's strategy for the set-aside is to start with the urban counties and then build out to the rural areas.
New York benefits from a productive state-academic partnership in the form of the New York State Psychiatric Institute,17 a state-funded research center affiliated with Columbia University. The institute is home to the Center for Practice Innovations, which under the direction of Lisa Dixon, the principal investigator for the Connection RAISE study, provides technical assistance and resources to states and providers seeking to adopt CSC. The Center for Practice Innovations has developed a training curriculum for the state's slightly modified version of the CSC model, both of which are called OnTrackNY. The state's strategy for using the set-aside funds has been informed by Dixon and her team at the Center for Practice Innovations from the beginning. The team continues to provide consulting and infrastructure support to the expanding OnTrackNY program network.
Outreach and Support Intervention Services (OASIS), the pre-existing ESMI program, was launched in 2005 with the financial support of two foundations that provided a three-year grant. The state began providing MHBG funding to the program after a request was made by a state representative who had heard about OASIS directly from one of its developers. Additionally, the state has partnered with OASIS in a number of mental health initiatives.
The state decided to limit eligibility for the ESMI programs to uninsured state residents with incomes less than 200 percent of the federal poverty level, thus excluding Medicaid recipients. Because most people under the age of 18 in Texas are covered by Medicaid, this decision also limited the proportion of people ages 18 and under being served by the programs. The SMHA made this decision in consultation with the Health and Human Services commission, the agency under which both the SMHA and the state's Medicaid agency operate. Although respondents reported that a number of factors entered into this decision, a key reason is the expansion of managed care in the state: while managed care organizations (MCOs) pay case rates for a given level of care, they do not have a rate for people with FEP; additionally, MCO rates for complex interventions are much lower than what the state is paying now.
The state has a sophisticated electronic web-based data system, the Clinical Management for Behavioral Health Services (CMBHS), that captures billing and clinical information from all publicly funded programs.
Virginia has sought to improve mental health services for transition-age populations since 2005, when the state passed legislation to improve the state's mental health crisis system. The Virginia Tech and Sandy Hook tragedies led to further reforms around civil commitment and temporary detention and prompted former Governor Bob McDonnell to create the Governor's School and Campus Safety Task Force in 2013. The task force developed a number of recommendations, including the enhancement of mental health services for transition-age individuals. Action on this recommendation was accelerated by another tragedy, the assault of Senator Creigh Deeds by his son, Gus, who eventually killed himself while in the midst of a psychotic episode, a tragedy that was widely viewed as the result of his release from a community hospital due to lack of psychiatric beds. These developments led to further mental health reform activity during the Virginia general assembly of January 2014 and, as a result, the 2014 session appropriated general funds to expand services for transition-age youth specifically. The MHBG 5-percent set-aside requirement was established around the same time and the SMHA decided to combine both funding streams to establish multiple ESMI programs in the state. Because the combined funds were sufficient to stand up seven programs with some funds left over, the agency decided to tap into additional MHBG funds to fund an eighth program.
Although the state has not expanded Medicaid, under the direction of the incumbent governor, the Medicaid agency launched the Governor's Access Plan (GAP) in January 2015 to improve access to certain Medicaid-covered behavioral health and primary care services for adults with SMI. (As defined by regulations, SMI determination requires illness of at least one year's duration.) Although eligibility criteria for coverage under the GAP are very strict, including an income of less than 60 percent of the federal poverty level, it is estimated that 20,000 residents would be eligible for Medicaid-covered services through the policy.
Washington's geographic proximity to Oregon has influenced the state's decisions regarding the design and implementation of the set-aside policy. (Oregon, the state where the EASA model was developed, introduced EASA-type programs for FEP in 2001, and they currently cover more than 90 percent of the population.) Washington has worked closely with Oregon-based experts throughout the policy implementation process.
No contextual information emerged during discussions that had the potential to influence the impact of the policy.
|TABLE 1. State-Specific Characteristics and Developments that May Influence Policy Implementation|
by December 2013b
|RAISE Site(s)||Tierc||1915(i) by
|State Characteristics and Developments
Affecting the Set-Aside Policy
|California||Yes||Yes||1||Yes||Highly decentralized state--set-aside funds were distributed among many counties|
|Colorado||No||Yes||3||Yes||Bureaucratic issues and staff changes delayed implementation|
|Connecticut||Yes||Yes||1||Yes||State is the site of 2 ESMI programs predating RAISE and the policy (STEP and IOL)|
|Idaho||No||No||2||Yes||Low public mental health spending; rural state|
|Iowa||No||Yes||3||Yes||State has two approved health home programs and will use MHBG funds to cover health home services for uninsured individuals|
|New York||Yes||Yes||1||No||Principal investigator for RAISE study (Connection) is based at Columbia University|
|North Carolina||Yes||No||2||No||Long-time state support for an ESMI program predating RAISE and the policy (OASIS)|
|Texas||No||No||2||Yesd||(i) Decision to only offer services to uninsured people excluded Medicaid recipients, including most individuals under 18 years old
(ii) State's sophisticated electronic web-based data system
|Virginia||No||No||2||No||(i) The Virginia Tech and Sandy Hook tragedies spawned the Governor School and Campus Safety Task Force in 2013, which recommended enhancing mental health services for young individuals. This development and the Deeds tragedy in November 2013 led to a large appropriation of state funds to expand mental health services for transition-age people in January 2014
(ii) The GAP implemented in January 2015 may improve access to health care for uninsured individuals with SMI
|Washington||No||No||3||No||Geographic proximity to Oregon, a state with a vibrant FEP tradition|
Level of Funding and Activities Funded by the Set-Aside
As shown in Table 2, the dollar amounts associated with the FY 2014 set-aside funds varied widely across states, from $186,765 in Iowa to more than $3 million in New York and California; this variation merely reflected the variable size of the MHBG across states. However, in California, the funds were divided into much smaller portions among the 57 county departments of mental health. Only a handful of states supplemented the set-aside funds with state general revenue (GR) or additional MHBG funds, and this was usually the result of specific legislative action, such as in Virginia, or the availability of state funds for similar purposes, as in California. States also varied in the scope of activities funded through the policy. Most states used the funds to support not only clinical services but also training of the clinical staff, evaluation of the ESMI program, or other activities (e.g., hiring external consultants for program design or development, augmenting other clinical programs, etc.). Only Tier 1 states with existing ESMI providers might have been able to allocate the set-aside funds to existing providers soon after the policy was implemented (FY 2014), but even those states engaged in planning and other infrastructure-development activities before they began disbursing funds to grantees. Hence, the initial set-aside allocation in several states was somewhat larger than the FY 2014 funds (i.e., if the allocation was made in FY 2015, it included the portion of FY 2014 funds not used for infrastructure-development activities and some portion of FY 2015 funds). Thus, some states, such as North Carolina, had more than one year's worth of funds at program launch.
Most Tier 2 and Tier 3 states used the set-aside funds to develop 1-2 ESMI programs; although North Carolina funded three programs, the state developed only two of those; the third program had been in operation since 2005. The large number of ESMI programs developed in Virginia was made possible only by a large infusion of state general revenue funds, $3.5 million for the state's FY 2015 (roughly matching the federal FY 2014) and $4 million for the state's FY 2016 (roughly matching the federal FY 2015), supplemented by a small amount of MHBG funds (see previous section).
The small number of programs allowed concentration of the funds and simplification of implementation and evaluation. New York, a Tier 1 state with more experience and resources, aimed to create a much larger number of new programs, using past experience as a guide. California is an outlier with its large number of programs, the direct result of the state's decision to manage the set-aside funds in the same decentralized fashion that the state manages the MHBG. (We were not able to determine the number of counties that are implementing ESMI programs because many county plans had not been submitted or reviewed at the time of the site visit.)
State Epidemiology of Relevance for ESMI Programs
Table 2 also shows the size of the population in the target age range of the ESMI programs (ages 15-24), and the size of the population in a similar age range (18-24), that is served by the public mental health system in each state. It is important to note that while the prevalence of SMI, as determined by survey assessments of representative samples, is quite similar across states, the proportion of the total population that is served by the public mental health system varies dramatically because of states' demographic characteristics or coverage generosity.
|TABLE 2. Size of Set-Aside Funds, Number of Grantees (ESMI Programs), and Target Populations by State|
|State||Set-Aside Funds ($)
|Number of Grantees to
Receive Set-Aside Funds
|Persons Ages 18-24
in SMHA (N)c
|Persons in SMHA as
Proportion of Total State
Population Ages 18-24d (%)
|NSDUH Estimates of
Percentage of SMI,
Ages 18-25, Past Year (%)e
|Connecticut||341,481||GR||2||8,265||2.39||4.03 (CI 3.12-5.20)|
|New York||3,090,758||No||10||76,597||3.84||3.94 (CI 3.30-4.69)|
|Idaho||127,740||No||1||2,422||1.55||5.13 (CI 4.06-6.46)|
|North Carolina||643,491||No||3||25,786||2.62||4.15 (CI 3.24-5.30)|
|Texas||1,829,838||No||2||36,650||1.35||3.49 (CI 2.90-4.18)|
|Virginia||570,327||GR, MHBG||8||10,790||1.30||3.88 (CI 2.99-5.02)|
|Wisconsin||369,000||Yes||1||11,472||2.05||4.61 (CI 3.63-5.85)|
|Colorado||341,481||No||1||11,124||2.15||4.41 (CI 3.45-5.61)|
|Iowa||186,765||No||2||13,027||4.08||4.27 (CI 3.31-5.48)|
|Nevada||227,500||No||1||3,077||1.21||4.37 (CI 3.37-5.66)|
|Washington||521,452||No||1||14,568||2.18||4.61 (CI 3.60-5.87)|
As shown in Table 3, states vary in their strategies for implementing the set-aside policy and in the models selected to serve individuals with ESMI. The clinical target population identified by states varies from only individuals with FEP (early, non-organic, non-affective, psychotic disorders), to those with any early psychotic disorder, to a still broader category of individuals with any ESMI. ESMI programs in most states serve youth and young adults, with the majority of programs serving individuals within or near the age range of 15-25. Maximum or expected program caseloads vary across states, and span from 15 to 151 individuals per team.
Implementation Strategy as Designed by the SMHA, the State Agency Administering the MHBG
We next explore the SMHAs' strategies for implementing the set-aside policy (see Table 3). We describe whether the strategy involved ESMI program expansion or development, and the number of ESMI programs (grantees) selected along with the states' rationales for their decisions. We also describe the states in terms of how prescriptive they were with regard to the model to be implemented (i.e., CSC versus the grantee's choice), target population (i.e., FEP-only, any early psychotic disorders, any ESMI, or the grantee's choice), training curriculum (i.e., OnTrackNY, NAVIGATE, or the grantee's choice), and uses of the set-aside funds (i.e., clinical services, training, program evaluation, other). We describe inclusion and exclusion criteria only if the SMHA was involved in their selection. Lastly, we describe the states' financing strategies with regard to the operations of ESMI programs for states that had a clearly delineated strategy; we note that the main difference among states with a clear strategy was whether there was an explicit expectation that the set-aside funds would be supplemented by third-party reimbursement when services were rendered to insured clients.
|TABLE 3. State Strategy and Grantee Description|
(Expand and/or Develop)
|Grantee(s) Contacted||Model Selected:b CSC,
Evidence-Based Variant, Other
|Target Population:c FEP,
Any Psychotic Disorder, Any ESMI
|Age Criteria||Maximum or Expected
Caseload per Program/Team
|California||Expand and develop||Alameda||PREP||FEP||16-24||60|
|Fresno's FOT Team||TIP||Any early psychotic disorder||16-28||30|
|Imperial||PIER||Any early psychotic disorder||15-25||15|
|San Luis Obispo||Other||Any person with mental illness experiencing a mental health crisis||18-22 (college students)||4-7 per week;
200 per year
|Connecticut||Expand||IOL||Other, Influenced by EPPIC||Any ESMI||16-24||75 current caseload;
~250 individuals per year
|STEP||STEP||FEP||16-35||Caseload currently ranges 100-200 patients per year|
|New York||Expand and develop||Northern Rivers Family Services||CSC||FEP||16-30||30|
|Idaho||St. Luke's Regional Health Center||CSC||Any early psychotic disorder||18-40||30|
|North Carolina||OASIS (including OASIS Carrboro and Wake STEP)||CSC||Any early psychotic disorder||16-30, with flexibility||Carrboro site: 151;
Wake STEP site: ~9
|Texas||Enhanced ePEP (Dallas Metrocare)||CSC||Any early psychotic disorder||15-30||30|
|MHMRA of Harris County||CSC||Any early psychotic disorder||15-30||30|
|Virginia||Alexandria CSB||CSC||Any early psychotic disorder||16-25, with flexibility||30|
|Fairfax-Falls Church CSB||CSC||Any early psychotic disorder||16-25, with flexibility||60|
|Highlands Community Services||CSC||Primary target: FEP;
Secondary target: individuals with second psychotic episode or with early bipolar disorder
|16-25, with flexibility||Target number of enrolled clients: 12-15d|
|Western Tidewater CSB||CSC||Primary target: any early psychotic disorder;
Secondary target: any ESMI
|16-25, with flexibility||30|
|Colorado||Jefferson Center for Mental Health||CSC||Any early psychotic disorder||15-25||No target caseload determined yet|
|Iowa||CMHC, Des Moines||CSC||FEP||15-30||Estimated statewide target population is 927, but no team caseload target yet|
|CMHC, Cedar Rapids||CSC||FEP||15-30||Estimated statewide target population is 927, but no team caseload target yet|
|Nevada||The Children's Cabinet of Northern Nevada||EASA||FEP||15-25||30|
|Washington||Central Washington Comprehensive Mental Health||EASA||FEP||15-25||N/A|
SMHA Strategy: Tier 1 States
As previously noted, California did not have a centrally determined strategy for implementing the set-aside policy; to the contrary, strategies were determined by each of the counties that submitted plans to use their potential allocation of set-aside funds. The state was not prescriptive in its implementation of the policy beyond requiring that use of the funds meet SAMHSA criteria. We were unable to determine what proportion of counties had their set-aside plans approved by the state and received funds to expand or develop ESMI programs. Of the four-county sample with which we held discussions,e two counties (Alameda and Fresno) applied their allocation of set-aside funding toward expansion of existing county-run ESMI programs, while the other two (Imperial and San Luis Obispo) pursued the development of new programs.
Connecticut's strategy for implementing the set-aside policy involved expanding the state's two existing ESMI programs, the STEP Program (based at Yale University, New Haven), and the IOL Program (in Hartford), because the amount of funding was only sufficient to expand pre-existing FEP programs. The state was not too prescriptive in its implementation of the policy, limiting itself to suggesting new ways for the existing programs to expand. We note that the Department of Children and Families (DCF), which resides within a separate state agency, receives 30 percent of the MHBG funds and also received a similar proportion of the set-aside funds. DCF decided to use its portion of the funding to enhance a program focusing on trauma identification and intervention.
New York's strategy for implementing the set-aside policy was to draw on the experience of the existing RAISE programs and expertise and on the empirical foundation of the CSC model and its implementation at the Center for Practice Innovations in New York City, to develop a network of ESMI programs across the state. The state was highly prescriptive in its implementation of the policy, including choice of model, target population, training curriculum, and uses of the funds. The state is implementing the OnTrackNY model, an adaptation of the CSC model used in the RAISE study, to offer services to state residents with FEP. Close to 90 percent of the set-aside funds were used for the four existing RAISE programs in the New York City region and several new FEP programs. The existing programs are being lightly modified to better adhere to the OnTrackNY model. The remaining funds are being used to support the Core Team, which is responsible for providing statewide support for the initiative, including training (which follows the OnTrackNY curriculum) and developing the evaluation infrastructure. Although the state estimates that 30 OnTrackNY teams ultimately will be needed to cover the entire state (based on an estimate of 20-30 cases per 100,000 population), the current plan is to establish six new OnTrackNY teams across the rest of the state. By the time of our site visit, there were six teams in infrastructure-development stage. The new OnTrackNY team based in Albany, which we visited, is administered by Northern Rivers, a multispecialty human services agency with a focus on services for children.
Inclusion/Exclusion Criteria. In keeping with the OnTrackNY model, the state determined that grantees would target individuals with FEP ages 16-30 within two years of their diagnosis.
Financing Strategy. The new programs will be fully funded through the set-aside funds for the first six months. During this time, Northern Rivers, in collaboration with the state, will examine the extent to which the services are billable to Medicaid or other insurance carriers and develop a sustainability model.
SMHA Strategy: Tier 2 States
Idaho's strategy for implementing the set-aside policy was to develop new ESMI services. The state selected St. Luke's Regional Health Center in Magic Valley-Twin Falls as the sole grantee. Using the same mechanism through which the SMHA enters into contractual relationships with MHBG grantees, the state awarded the contract for the ESMI program to St. Luke's without competition. This was based on the state's assessment that St. Luke's had the capacity, as well as the population needs and resources available within the region, to implement an ESMI program successfully. Idaho was highly prescriptive in terms of choice of model, method to estimate the target population, training curriculum, and uses of the funds. The state decided to implement the CSC model. The SMHA planned to use the set-aside funds to contract for a needs assessment to identify and quantify the potential target population for the program, fund a nine-month pilot CSC program, and fund an evidence-based training program, data analysis, and outcomes reports.
Financing Strategy. The state is delivering the set-aside funds in fixed monthly amounts to compensate for services rendered. While some clients may be more costly than others, St. Luke's is responsible for covering the cost of services with the monthly set-aside allocations. Additionally, the state expects that St. Luke's will leverage clients' insurance for reimbursable services, and will not turn individuals away based on financial need.
North Carolina has had an early intervention program for people with early psychosis based on the EPPIC model called OASIS since 2005. Although its developers stated that the model shares some critical components with CSC, they noted that OASIS places a strong emphasis on general health and wellness as a fundamental aspect to the recovery process. Furthermore, the program does not discharge patients, thus carrying a large caseload comprising clients in varying stages of acuity and recovery. The original program located in Carrboro in Orange County, was initially funded with a three-year grant from KB Reynolds and the Duke Endowment, but it has received MHBG funding over the years. North Carolina's strategy for implementing the set-aside policy was to provide further support to the OASIS program, as well as develop two new ESMI programs. The SMHA issued an invitation to apply for the set-aside funds, expecting that the state would contract with OASIS and additional providers. After reviewing the proposals, the SMHA selected two new providers: a newly established OASIS site based in Raleigh in Wake County (the Wake Schizophrenia Treatment and Evaluation Program, or Wake STEP, managed by the Alliance Behavioral Healthcare MCO), and a provider called RHA Behavioral Health Services based in Wilmington in New Hanover County (managed by the CoastalCare MCO). In selecting the new clinical sites, the state evaluated the size of youth population and tried to not focus exclusively on urban areas. Because OASIS has a good track record of outreach to other counties, they expected a broad geographic reach for the two OASIS programs. Although the Wilmington area is quite populous and there is a university nearby, their expectation is that RHA Behavioral Health Services would also reach the rural areas in that part of the state.
The state was only moderately prescriptive in its implementation of the policy. North Carolina decided to implement the CSC model and, as part of this decision, the state provided funding to the original OASIS program to help it become more CSC-like. North Carolina's decision to adopt CSC as opposed to OASIS was influenced by the state's understanding that the models were quite similar and the desire to adhere to the SAMHSA guidance as closely as possible. The state distributed more or less equal allocations to the three programs, but the funds allocated to the original OASIS-Carrboro Program included funds for technical assistance and the evaluation component (the state recommended several data elements for consideration but was not otherwise prescriptive). Funds to the OASIS-Carrboro program were also used to hire staff, some of whom had been previously available because of their university connections but in an inconsistent fashion. The state was not prescriptive in terms of target population (i.e., whether FEP or any early psychotic disorder) or regarding training curriculum--although OASIS has developed its own training curriculum with set-aside funding, the RHA Behavioral Health Services site has also used OnTrackNY training resources.
Financing Strategy. The portion of the funds allocated to clinical services will be primarily used to support staff salaries; the balance of the funds, approximately 25 percent, will be used for program supplies, transportation, logistics, and covering non-reimbursable services and the uninsured/underinsured. While the SMHA anticipates that the grantees will identify revenue sources and work toward obtaining reimbursement from third-party payers, state respondents reported that their priority at the moment is for providers "to work on getting the practice model in place firmly...getting the administrative components up and running and the staff trained." The original OASIS program captures private insurance for some of their services, as illustrated by a breakdown of recent funding streams: 60 percent private, 23 percent Medicaid, 2 percent Medicare, and 15 percent with no insurance.
Texas's strategy for implementing the set-aside policy was to develop new ESMI services. The state was highly prescriptive in its implementation of the policy, including choice of model, target population, training curriculum, and uses of the funds. Texas is implementing the CSC model to offer services to individuals with any early psychotic disorder. The state selected two providers as grantees, with the intention of funding two teams per grantee: the Mental Health and Mental Retardation Authority (MHMRA) of Harris County, a Houston-based Local Mental Health Authority (LMHA), and Metrocare Pathways Clinic, a Dallas-based North Texas Behavioral Health Authority (NTBHA) provider.g The SMHA used the following criteria to select grantees: (1) location in urban areas to maximize the chances of referrals (citing the recommendation in the RAISE literature that a fully dedicated team should cover a population of about 500,000 to be sustainable); (2) close proximity to academic institutions capable of leading the evaluation component; (3) ability to compare the CSC implementation process for an LMHA-based provider (operating through a regular state performance contract) and a provider operating under ValueOptions, the carve-out managing care for NTBHA; and (4) previous experience providing ESMI services for transition-age youth (Metrocare-Dallas). The state specified target caseloads (30 clients per team) and staffing (six-member teams, including a peer specialist). The SMHA contracted with the Center for Practice Innovations to provide OnTrackNY training to the new teams using set-aside funds, and required that the grantees use the set-aside funds to subcontract with academic institutions to evaluate the newly developed programs.
Inclusion/Exclusion Criteria. The state required that the grantees target individuals with any early psychotic disorder ages 15-30 within two years of their diagnosis.h As previously mentioned, Texas also determined that the set-aside funded services would only be available to uninsured state residents with incomes less than 200 percent of the federal poverty level.
Financing Strategy. The state had the expectation that the grantees would tap into the organizations "with large revenue streams" that they are nested within (i.e., MHMRA of Harris County and Metrocare Pathways Clinic) for financial assistance with staffing, thus permitting them to use a larger portion of the set-aside funds for their operations. Although grantees receive one-twelfth of the budget at the end of each month to ensure they can operate, they receive the balance only if they deliver a predetermined number of treatment hours, a strategy designed to promote productivity ("per diem process"). Although the SMHA had interpreted from the SAMHSA guidance that the set-aside funds could not be used to cover outreach activities and as a result, they are not covered by the "per diem process," respondents thought that the generosity of the per diem rate and the fact that the teams were not at capacity yet would provide adequate incentives for the teams to engage in outreach and client recruitment activities.
Virginia's strategy for implementing the set-aside policy was to develop new ESMI services. In July 2014, the SMHA released a request for proposals to the state's 40 Community Service Boards (CSBs)i stating that "the FY 2015 state budget [~FY 2014 federal budget] and the federal MHBG include funding to support the development and expansion of services for young adults ages 16-25 experiencing serious behavioral health conditions, including substance use/abuse and the initial onset of psychosis. Through a combination of these funds, [the SMHA] will have approximately $4 million available in FY 2015 and beyond to support multiple awards for demonstration initiatives to develop services for this population."j The state was moderately prescriptive in its implementation of the policy, expressing a preference for the CSC model in its request for proposals, and identifying a broadly defined target population that includes people with non-psychotic disorders. While requiring that the grantees "measure, monitor and report fidelity to their chosen modality and outcomes of services provided," the state was not prescriptive about the evaluation plan and it did not allocate separate set-aside funds to the evaluation. The state successfully applied for additional SAMHSA technical assistance funds to make OnTrackNY trainings provided by the Center for Practice Innovations available to grantees.
Fourteen of the 18 applications from five of the seven regions in the state proposed a CSC model. Because the review committee wanted to ensure adequate geographic representation of ESMI services across the state, the top-scoring application from each of the five regions was selected along with the three top-scoring CSBs from the remainder of the applications, thus selecting eight grantees.
Financing Strategy. Virginia did not have an expectation that the grantees would need to bill third-party payers or use additional funding sources to operate the programs. In its request for proposals, the state did ask prospective grantees to "describe any existing funds or in-kind support (that would be allocated) to the proposed services and any other anticipated revenues (e.g., Medicaid) that would be generated," which several of the successful applicants did.
Wisconsin's strategy for implementing the set-aside policy was to develop new ESMI services. The SMHA determined that the annual set-aside fund allocation for Wisconsin would be sufficient to support one ESMI program providing services for a population of approximately 500,000-525,000, based on recommendations from the RAISE literature. The state released a request for proposals in August 2014 with the intention to launch the winning contract on October 1, 2014, and start service delivery in early 2015. The SMHA required that applicants "have, or show they can develop...identified expertise in providing care to youth who are experiencing psychotic illness; wellness, recovery, and resilience orientation; access to inpatient hospital care; linkages with community resources and outreach capabilities; strong psychiatric supervision and clinical leadership; commitment to hiring individuals with lived experience; and ability to provide data."k Furthermore, the SMHA required that applicants have a certified psychosocial rehabilitation program "to serve as a foundation for the CSC program."l The state selected Journey Mental Health Center (JMHC), a large and established provider located in an urban region (Madison in Dane County) as the sole grantee.
The state was highly prescriptive in its implementation of the policy, including choice of model, target population, training curriculum, and uses of the funds. The state decided to implement the CSC model to serve individuals with FEP. The program was expected to maintain a caseload size "of 30 clients or less to ensure adequate time to maintain fidelity to the model services, develop relationships, and provide outreach to the participants," and provide time-limited services (2-5 years). The state decided that the set-aside funds would be used solely to implement the CSC. The SMHA determined that oversight of program fidelity would be done by the agency's contract manager, whose salary is paid for by the larger MHBG. The outcomes evaluation component will be designed and overseen by a researcher affiliated with the University of Wisconsin-Madison who has volunteered to undertake this activity free of charge. Using SAMHSA funds, the state recently contracted with Center for Practice Innovations to provide technical assistance to both the grantee and a Milwaukee-based provider that had failed in its bid for the funds but decided to set up a CSC program using local funds.
Inclusion/Exclusion Criteria. The SMHA determined that the grantee would target individuals with FEP ages 15-25 within three years of symptom onset.
Financing Strategy. The SMHA expects that the grantee will bill private insurance and Medicaid for all billable services, including psychosocial rehabilitation services, to ensure program sustainability.
SMHA Strategy: Tier 3 States
The planning for the implementation of set-aside policy was conducted with input from the state's Behavioral Health Planning and Advisory Council and Dr. Lisa Dixon and her team at the Center for Practice Innovations. The Center for Practice Innovations has been contracted to provide consultation and technical assistance for implementation and evaluation of the program. The SMHA decided to implement the CSC model and fund one grantee to be selected through a competitive bidding process. The state's expectation was that the experience from the funded program would inform the future expansion of the model, with the goal of better meeting the needs of transition-age youth across the state. The SMHA had anticipated receiving responses from CMHCs, other specialty mental health clinics, and possibly the University of Colorado. The request for proposals stated that the successful grantee would implement a high-fidelity CSC program with the goal of improving outcomes for youth with early psychosis, including affective disorders (i.e., any early psychotic disorder). The state required that applicants commit to adhering to existing regulations if they are not already an approved CMHC or a specialty clinic. The SMHA also expressed a preference for proposals that would build on existing services targeting youth ages 15-25 with SMI. Colorado ultimately selected Jefferson Center for Mental Health as the state's sole grantee. The provider is located just west of Denver in Jefferson County, one of the most populous counties in the state. The state envisions that the program will also serve two neighboring mountain counties, slightly more rural, to meet caseload requirements.
Thus far, Colorado has been moderately prescriptive in its implementation of the policy, having selected both the model and the target population.
Inclusion/Exclusion Criteria. The SMHA determined that the grantee would target individuals with any early psychotic disorder ages 15-25 within three years of symptom onset.
Iowa's initial strategy for implementing the set-aside policy was to allocate all the funding to an existing RAISE study site with the goal of expanding the program. However, the RAISE team dissolved sometime after the policy was implemented, so the SMHA decided to develop new ESMI programs. The state decided to implement the CSC model to serve individuals with FEP, and fund two grantees whose training could be combined to achieve economies of scale. The SMHA only received two applications in response to their request for proposals. Potentially eligible providers in rural areas relayed to the SMHA that they would not have the resources to develop and deliver CSC services with adequate fidelity. Contracts were awarded in April 2015 to two CMHCs. The state has been highly prescriptive in its implementation of the policy. The state selected the model, target population, and training curriculum (NAVIGATE) for the CSC programs.
Inclusion/Exclusion Criteria. The SMHA determined that the grantees would target youth and young adults ages 15-30 experiencing FEP.
Nevada's strategy for implementing the set-aside policy was to develop a new ESMI program modeled after the EASA model. During the planning phase, the state has worked closely with its chosen provider, The Children's Cabinet, a large child social services provider in Reno, and consultants from Oregon's EASA/FEP program to develop a detailed list of services that will be included in the program, a listing of team members and their roles and responsibilities, and an organizational chart showing the institutional location of each team member. The program is scheduled to start operating during the second half of 2015 in Reno and then expand to Las Vegas at a later point in time, after gaining initial implementation experience. The Children's Cabinet is the largest child social services provider in the Reno area and was well positioned to establish working relationships with the full range of providers required by a FEP program.
Inclusion/Exclusion Criteria. The SMHA determined that the grantee would target individuals with FEP ages 15-25 who were within 18 months of symptom onset. Individuals will be excluded if they have received treatment for psychotic symptoms for more than three months; have a significant medical condition that prevents participation in services; or are incarcerated. These criteria will be followed during the implementation of the Reno program and potentially adjusted according to that initial experience.
Financing Strategy. The state expects that state-funded clinical staff will play a large role in the FEP program--this strategy would allow the program to have access to medical staff without having to support them financially. The state also expects that the FEP program will actively seek reimbursement from third-party payers.
Washington's strategy for implementing the set-aside policy was to allocate the bulk of the fundingo to develop a new ESMI program modeled after the EASA model and conceived as a demonstration project that will inform future expansion of ESMI services to other regions in the state. The state has worked closely with EASA experts based in Oregonp throughout the program development process.
The SMHA selected Central Washington Comprehensive Mental Health, a large multispecialty human services provider, as its sole grantee; its ESMI program is called The New Journeys program. The provider is located in a relatively rural area of the state, Yakima County, with a population size of 250,000. There were several reasons behind their choice of provider:
The provider already offers the full range of services that are needed to implement a CSC program--including a broad range of health care professionals, many of whom are bilingual, and an intensive case management program for SMI individuals (Wraparound with Intensive Services); moreover, services are integrated into a single electronic health record (EHR) system that will greatly facilitate care coordination.
The agency has strong community connections, including connections with law enforcement, that will facilitate outreach and naturally configure a network of providers to support program clients.
The agency has experience billing both Medicaid and private insurers for services, whereas many agencies would require extensive technical support to develop that capacity, which is critical to the long-term sustainability of the program.
The state has been highly prescriptive in its implementation of the policy, including choice of model, target population, training curriculum, and uses of the funds. The SMHA selected the EASA model based on its assessment that the model has proven to be successful in improving outcomes for youth and young adults statewide in Oregon. The state determined that, in keeping with the EASA model, the target population would be restricted to individuals with FEP. Moreover, it was determined that the program would be time-limited, with two years for most individuals, and a possibility of extending the program to three years for those in need. Washington has contracts in place for clinical training, educational presentations for providers on FEP programs, and consultation for implementation of the EASA model.
Inclusion/Exclusion Criteria. The SMHA determined that the grantee would target young individuals with FEP.q While the program is located in Yakima County, once it has established a good foundation in that county, the catchment area will be extended to surrounding counties.
Implementation Strategy as Designed by the ESMI Programs
Next, we describe in detail the ESMI programs that we held discussions with (see Table 3). For states where programs had ample latitude in terms of model, training curriculum, target population, or uses of the funds, we describe those domains and describe program status--including whether operational, and current caseload, if so. For states where we were able to collect information on partnerships and collaborations, we also discuss this component of program implementation. Programs' partnerships and collaborations are important because they provide a valuable mechanism for outreach to potential referral sources,r which is also described in this section whenever the information was captured. (Partnerships and collaborations are also important for design and implementation of the evaluation strategy, which is described in the next section.)
Program Strategy: Tier 1 States
Alameda County used the set-aside funds to enhance a pre-existing FEP program based on the PREP model24 and located within a larger Transitional Age Youth (TAY) program. The PREP Program assists up to 60 youth and young adults with recent onset of non-affective psychosis by offering direct and accessible clinical services. Concerns about sustainability of the set-aside funding led county officials to use the funds for training and capacity-building within the PREP program, thought to be very similar in content to CSC, rather than change the model. The training is provided by the Felton Institute, a private non-profit social service provider and developer of the PREP program based in San Francisco, and it focuses on strengthening the clinical skills needed to successfully implement the model. The training covers methods for early identification of psychosis, differential diagnosis of SMIs, and Cognitive Behavioral Therapy for Psychosis (CBTp). The training consists of initial sessions held at the institute and follow-up supervision and feedback provided over six months following the in-person sessions.
Partnerships and Collaborations. The program works closely with clinical partners at University of California-San Francisco (UCSF) and training partners at the Felton Institute. The program is staffed with UCSF-based psychiatrists and the Felton Institute provides training to program staff in the clinical components of the model and consultation to the program regarding evaluation methods.
Fresno County used the set-aside funds to enhance the First Onset Psychosis Team (FOT), a pre-existing program that had been in operation since 2010 supported with funds from the Mental Health Services Acts and Medicaid reimbursements. The FOT carries a caseload of about 30 clients between 16 and 28 years of age with any early psychotic disorders. The program consists of six clinicians and four case managers who divide their time between this program and a more general program for at-risk TAY. The FOT uses a Transitions to Independence Process (TIP) model,25, t described as CSC-like, to provide a CBTp-based treatment that also includes psychopharmacology. The set-aside funds are being used to enhance the program and augment it with CSC components. Specifically, the funds support: (1) CBTp training provided by the Felton Institute and targeted to team psychologists; (2) peer support training in the TIP model provided by a large provider of community mental health services located in Long Beach (STARS Behavioral Health Group); and (3) consultant-led training in the CSC model.
Partnerships and Collaborations. The program had prior partnerships with hospitals, crisis centers, and emergency rooms in the county, which are currently their main referral source.
Outreach to Potential Referral Sources. Client recruitment occurs in multiple settings across the counties, through outreach by the team and referrals from collaborating providers. The team has positive relationships with hospitals, crisis centers, and emergency departments in the county and those sites have been the largest sources of patients to date. Team members visit potential clients during psychiatric hospitalizations. Although the program is about to reach capacity, it is extending the referral network beyond health care institutions to local schools and colleges.
Imperial County used the set-aside funds to augment the capability of its existing youth services program, called Transitional Engagement Supportive Services (TESS), with the goal of addressing the needs of adolescents and young adults experiencing ESMI. The TESS program provides a host of supports to individuals being discharged from the hospital or other crisis intervention services for mental illness to assist with their reintegration into the community.u
The set-aside funds will be used to strengthen a specialized component of the TESS program that will connect adults ages 15-25 who are at risk for or currently experiencing the first episode of any psychotic disorder with the county's existing Full Service Partnership (FSP) program.v The county described this strategy as an efficient way to improve services for this population because the TESS program already provides services to young adults.
To provide the connection to the FSP program, the county adopted the PIER model,w which was recommended by its creator, William McFarlane, and had been successfully implemented in other California counties. County officials decided to augment the PIER model with a family involvement component they thought would be a good fit for their population, a large proportion of which is Latino. The PIER model will be implemented largely by tapping into existing FSP resources and using the set-aside funds to support additional clinical staff time. The set-aside funds will enable the PIER program to screen for recent onset psychosis and expand outreach capacity through presentations in schools and education about psychosis for law enforcement. Respondents described that part of the PIER model includes a preliminary phase (approximately 3-4 months) of outreach and community education, followed by initiating program referrals. The program completed its outreach phase and identified three clients at the time of our discussion in April 2015. The county anticipates that the program will ultimately serve 15 clients in the set-aside target population. The program has been operational since January 2015.
Outreach to Potential Referral Sources. Program staff plan to provide presentations to law enforcement and at schools to improve awareness of psychosis and as a client recruitment strategy.
San Luis Obispo County
Given that the county did not have a pre-existing ESMI program, coupled with the small size of the set-aside funding allocation, the county decided to use the funds to support the development of a mental health crisis intervention program within the residence halls of the California Polytechnic State University (Cal Poly), a university with a large and growing residential student body. The plan was developed in collaboration with Cal Poly's health staff. The program will work with the county's mobile crisis team to link individuals to more-intensive services, such as hospitalization. The county also hopes to introduce a student mental health liaison at the university using Mental Health Service Act funds, but that decision has not been finalized. The target population includes individuals with any mental illness experiencing a mental health crisis and those with co-occurring substance use disorders. At the time of our telephone discussions, the overall structure of the program had been finalized and the county was about to start advertising positions with the goal of being fully staffed at the opening of the 2015 fall semester.x
Partnerships and Collaborations. The program has a strong partnership with Cal Poly's administration and police department. Additionally, the program coordinates with the county's mobile crisis provider, which is available 24 hours a day, seven days a week, for immediate response including involuntary psychiatric holds.
Institute of Living
The IOL program, based in Hartford, was launched in 2004 as a "grass-roots" effort designed to serve individuals ages 16-24 with ESMI, by treating both the individual and the family and providing cognitive remediation with a heavy emphasis on family education and support over time. Services were developed in conjunction with the International Early Psychosis Association and included influences from Australia's EPPIC program24 and research based in Norway; additionally, William McFarlane was brought in as a consultant. The program provides services to both individuals and their families at no cost. The program provides a range of services for young adults, including an intensive outpatient program, a dual-diagnosis program, and a Lesbian, Gay, Bisexual, Transgender, and Queer program. At the request of the SMHA, the IOL program used the set-aside funds to augment its vocational services (including job readiness and payment for persons taking part in supervised work assignments), outreach, and peer services. Thus, the set-aside funds are being used to expand ("enrich," as described by a respondent) the services that were already being provided by the IOL program.
While the IOL program in 2004 consisted of 12 hours of staff time per month, the program now has 12 full-time staff supported through MHBG funds and from billing for intensive outpatient program services and other services. The program's current caseload is approximately 75 individuals. The majority of these clients are receiving early intervention services, which consist of three encounters a week per person. The program serves 150-200 clients per year.
Financing Strategy. In addition to actively billing third-party payers for billable services and the recent infusion of set-aside funds, the program taps into several funding sources. For example, funding for client engagement services is provided by private, state, and MHBG funding, and costs associated with family-oriented services are covered by a foundation.
Specialized Treatment Early in Psychosis
The STEP program, affiliated with Yale University, New Haven, was set up in the early 1960s to help transition people from state mental hospitals back into the community,24 following similar models that were being implemented in the United Kingdom, Australia, and Northern Europe. The model has undergone revisions, most recently in 2013, mainly related to the psychosocial interventions delivered as part of the program. The program serves individuals with FEP ages 16-35. The MHBG set-aside funds are being used to enhance the vocational services component of the STEP program.
The clinician-patient ratio is 1:30 or 1:40, with mostly office-based services mainly due to lack of outreach-enabling resources. The maximum caseload at the STEP Program has fluctuated since its inception, and currently ranges from 100 to 200 patients per year.
Financing Strategy. The version of the model established in 2006 lacked specific funding sources, which is notable because similar programs had only existed in countries with universal access to health care.
Outreach to Potential Referral Sources. Approximately 50 percent of the referrals come from a private non-profit inpatient psychiatric hospital and more than a third of the referrals are from triage services at the Connecticut Mental Health Center. Other referral sources are a walk-in clinic, private providers, and self-referrals following an advertising campaign.
Partnerships and Collaborations (IOL and STEP). Both programs reported they rely on the LMHA for certain components of their ESMI models (crisis services, intensive case management).
At the time of our site visit, the new programs were in the very early stages of implementation, either still in contract negotiations with the state or just beginning to hire staff. However, Northern Rivers Family Services, the OnTrackNY provider based in Albany with which we held discussions, is already receiving phone calls with potential referrals from the community. Although the program is in its very early stages and several months from enrolling its first client, Northern Rivers staff has begun serving some of the newly referred individuals in their existing mental health clinic until the OnTrackNY program is operational. The program will draw from residents of four surrounding counties (Albany, Schenectady, Rensselaer, and Saratoga) to build up its target caseload of 30 clients.
Partnerships and Collaborations. Northern Rivers has been developing methods for evaluating the FEP program with experts affiliated with Columbia University.
Outreach to Potential Referral Sources. In keeping with the OnTrackNY model, program staff plans to develop close relationships with providers in the region to establish a referral network. The plan is to focus initially on emergency departments and psychiatric inpatient units where young people experiencing FEP are likely to be found. Once the program is more-established, the plan is to expand outreach to medical practices, schools, probation officers, peer-run services (such as People Inc., a local peer-run service agency), school districts, and community organizations in an effort to identify youth in distress and to educate potential referral sources about the recovery-based approach.
Program Strategy: Tier 2 States
St. Luke's Regional Health Center, the sole grantee, selected the RAISE (CSC) model based on a review of the evidence and the grantee's assessment of what was feasible to implement in the state. The decision was also helped by participation in webinars provided by experts based in Oregon, Maryland, and New York, as well as consultation with the Center for Practice Innovations. Respondents reported that St. Luke's implemented the CSC model in a way that was largely consistent with OnTrackNY; the program serves individuals with any early psychotic disorder. As above, the state had prescribed that the set-aside funds would be used to fund a needs assessment (which included a chart review), CSC trainings for staff, implementation of a nine-month clinical pilot program serving the population identified in the needs assessment, and an outcomes evaluation.
The program targets individuals ages 18-40 with any early psychotic disorder within two years of their diagnosis. The higher age cap resulted from the needs assessment, which convinced the program developers that such a modification was needed to ensure sufficient number of program participants. The program serves an eight-county region, the catchment area of the local Department of Health and Welfare. The area is approximately 12,000 square miles and including a total population of approximately 185,000 people. At the time of our site visit (April 2015), the fully staffed program had been in operation for approximately two months.
Partnerships and Collaborations. St. Luke's is working with local partners to enable client access to transportation, housing services, and vocational rehabilitation.
Outreach to Potential Referral Sources. The program conducts outreach activities at the inpatient unit of the local psychiatric adult hospital and outpatient clinical settings, including St. Luke's own outpatient behavioral health clinic.
While both OASIS programs target individuals ages 16-30 with any early psychotic disorders, they have flexibility on both ends of the age spectrum and have an assertive policy of not excluding those with co-occurring conditions, such as developmental disabilities, borderline personality disorder, or substance use disorder. The catchment area extends to within 1.5 hours driving distance of either site location.
The OASIS program based in Carrboro in Orange County, in operation since 2005, receives 4-5 referrals monthly. While approximately 20 percent of the clients receive intensive treatment services at a given time, a majority of clients have stabilized and are not currently receiving intensive services. Approximately 10 percent of the client population has more-chronic symptoms. The program's current caseload is 151 clients, but more than three-quarters of them (N=113) have been in the program for more than two years, receiving services at the Carrboro site. Although many of these clients would not be considered to have an early psychotic disorder, as previously noted (see State Strategy, Tier 2 States), the OASIS model differs from CSC in that the program does not discharge patients. The OASIS Program based in Raleigh in Wake County (Wake STEP) has been in operation for a shorter period of time but has expanded rapidly and is now receiving many referrals. As of August 2015, the caseload of the Raleigh site was approximately nine clients.
Partnerships and Collaborations. The developers of the OASIS program are affiliated with the University of North Carolina (UNC) Center for Excellence in Community Mental Health, at the Department of Psychiatry at UNC's School of Medicine. Thus, both OASIS programs have a close relationship with UNC. OASIS also has a close relationship with Duke University. Other organizations and agencies with which OASIS has developed strong partnerships include the Local Management Entities-MCOs for the region, crisis intervention teams, law enforcement, community/family resource courts, and area health education centers.
Outreach to Potential Referral Sources. OASIS conducts regular outreach to the access units within the Local Management Entities-MCOs, as well as local emergency departments and hospitals, small medical practices, high schools, and colleges/universities (UNC and Duke University are an important source of referrals). OASIS is also working with Alliance, the Local Management Entity-MCO in Wake County, to educate high school counselors and improve early recognition of psychotic symptoms in high school settings. OASIS also promotes awareness about the program through periodic mailings and web-based trainings held for locally based clinicians (including pediatricians, family practice physicians, psychologists, family medicine), which they anticipate expanding to clinicians statewide.
As previously mentioned, the state has two newly developed CSC programs, one based at MHMRA of Harris County, a Houston-based LMHA, and the other based at Metrocare Pathways Clinic, a Dallas-based NTBHA provider. While both programs closely adhere to the CSC model, the Dallas Metrocare CSC program (Enhanced Program for Early Psychosis or ePEP) also provides cognitive remediation. The CSC program based at MHMRA of Harris County was able to become operational sooner than the Dallas Metrocare CSC program. This was partly a result of the MHMRA repurposing existing staff, which the Dallas Metrocare CSC program was unable to do because the agency was operating at capacity.
CSC Program at MHMRA of Harris County
The CSC program based at MHMRA of Harris County began recruiting patients in November 2014. At the time of our site visit, the program had two fully staffed teams with a total caseload of 61 clients.
Partnerships and Collaborations. The agency has a close partnership with the University of Texas (UT) Medical School at Houston. Researchers from UT's Department of Psychiatry and Behavioral Sciences are leading the subcontract to design and implement the evaluation of the newly developed program. The existing data sharing agreement between UT and the agency will greatly facilitate the evaluation; moreover, UT and its medical center are an important source of referrals for the program.
Outreach to Potential Referral Sources. In the initial phase of client recruitment, the CSC program restricted its efforts to reviewing lists of current LMHA clients to identify potential CSC clients based on age and diagnosis. The clinic also received some of its current clients through the Harris County Psychiatry Center (a local county hospital) and the Harris County eligibility center (central location for individuals in the community to come and request LMHA services). The clinic plans to take advantage of previously established co-located programs--LMHA clinical staff provide child services near schools--for recruitment. The deputy director for MHMRA of Harris County has advertised the services to all the other clinics in the area, and the CSC program has gotten referrals from those clinics as well.
Enhanced Program for Early Psychosis (Dallas CSC program)
The ePEP, based at the Metrocare Pathways Clinic in Dallas, began recruiting patients in February 2015. At the time of our in-depth telephone discussion, the program had one fully staffed team and another with most of the staff in place, and a total caseload of 36 clients.
Partnerships and Collaborations. The agency has a close partnership with UT and its Southwestern Medical Center located in Dallas, where the lead investigator for the CSC evaluation is a professor of psychiatry. ValueOptions, the carve-out managing the care provided by ePEP and all behavioral health providers in the NTBHA, is an important partner in the program implementation, as it provides the CSC with lists of potential clients in the program's catchment area and it also collects clinical and cost data that may be used for the evaluation component.
Outreach to Potential Referral Sources. Thus far, client recruitment efforts have included advertising CSC services internally (Metrocare Pathways Clinic, which has several sites) and at local hospitals (Timberlawn, Green Oaks, Terrell State). Program staff also regularly search through lists of potential clients provided by ValueOptions.
As already mentioned, Virginia developed eight new CSC programs. Although the state indicated that CSC services could be offered to a broadly defined ESMI population ages 16-25 that included people with non-psychotic disorders, several programs were more restrictive in their clinical target population. However, they expressed flexibility with regard to age, accepting younger and older clients.
The expected start date for the CSC programs was October 1, 2014, and the programs were expected to be fully operational by June 30, 2015. Several programs were already operational before that date. Only one grantee was unable to meet that expectation because it had planned to do a sole-source contract with a community provider but the county had instructed the grantee to do a public solicitation of services.
Seven programs are using the OnTrackNY training curriculum, one CSC program (Highlands CSB) is exclusively using the NAVIGATE training curriculum because NAVIGATE seemed better aligned with the program's rural setting, and another program (Western Tidewater CSB) is using a combination of both curricula.
While the state did not have an expectation that the grantees would need to tap into additional funding sources to operate, several CSC programs were expecting to bill private insurance for billable services.
Alexandria Department of Community and Human Services (DCHS)/CSB CSC Program
In addition to all CSC components, the Transitioning Adults into Living Successfully (TRAILS) CSC program in Alexandria provides permanent supported housing to approximately ten clients per year through a contract with a private, non-profit community provider. The program is located in the West End Wellness Center, a psychosocial rehabilitation center located within the CSB's Center for Adult Services, but it will work in close collaboration with the CSB's Center for Children and Families.
The program's current target population is individuals with any early psychotic disorder within five years of symptom onset. However, program staff is considering stricter eligibility criteria to be better aligned with the CSC model--that is, include only FEP of onset within two years of intake. While the target caseload is 30 clients, at the time of the site visit, the CSC program had enrolled four clients and had 7-10 referrals.
Partnerships and Collaborations. The Alexandria CSB has a close relationship with a state hospital within its catchment area, the Northern Virginia Mental Health Institute. The agency also partners and collaborates with the local public schools; specifically, the school social workers--CSB staff regularly attends individualized educational program meetings and other school activities, and a CSB-affiliated mental health professional is among the staff at the TC Williams High School's ninth-grade Teen Wellness Center.y Lastly, the CSB has a partnership with the court system and deploys mental health professionals in the court services unit.
Outreach to Potential Referral Sources. At the time of our site visit, most of the outreach efforts had targeted emergency services and the Northern Virginia Mental Health Institute. The CSB also has reached out to the local Family Assessment Planning Teamz to discuss potential outreach strategies. The agency also plans to seek referrals from the local public and private high schools and it will reach out to student health services at local community colleges and nearby universities (e.g., George Washington University, George Mason University, Howard University). The CSB plans to leverage its partnership with the local court services unit for referrals.
Fairfax-Falls Church CSB CSC Program
The Fairfax-Falls Church CSB implemented a CSC program in collaboration with the Fairfax Psychiatric Rehabilitation Services (PRS). The target population for the Fairfax-Falls Church CSC program is individuals diagnosed with any early psychotic disorders within two years of diagnosis. The program's rationale for selecting a relatively broad diagnostic category was that they did not want to exclude clients who might benefit from CSC services. However, the program will consider focusing on FEP and excluding affective psychotic disorders, given that the evidence base is scarcer for the latter group, if the caseload grows rapidly and they reach capacity too soon.
Although program staff had initially planned a target caseload of 80 clients within the first year, they decided to reduce that number to 60, as a result of knowledge gained during an OnTrackNY training. At the time of our site visit, the CSC program had enrolled seven clients and had 38 referrals.
Financing Strategy. The CSC program expects that it will be able to bill private insurance for billable services as a result of many young people being covered under their parents' insurance and the "level of affluence" in the county.
Partnerships and Collaborations. The Fairfax-Falls Church CSB is partnering with the Fairfax PRS, to develop its CSC program. The CSB and PRS have had a close partnership for about 50 years. The CSB has contracted with PRS to provide psychosocial day programs and vocational services for CSB clients.
Outreach to Potential Referral Sources. Program staff have reached out to other CSB programs, crisis services, the main hospital system in Fairfax, public high schools, the Northern Virginia Community College and two local universities (Marymount University and George Mason University), a local jail, and the Offender Aid and Restoration program,aa among other social service agencies. Program staff is looking for listservs listing local pediatricians and is considering outreach strategies to target local CMHCs and federally qualified health centers. Finally, the CSC program advertises its services through its own website (http://turningpointcsc.org/).
Highlands CSB CSC Program
The target population for the Highlands CSB CSC program is individuals with FEP within two years of symptom onset. However, if the program does not grow too rapidly, the CSB will consider expanding the clinical eligibility criteria to serve clients in their second psychotic episode or with early bipolar disorder.
At the time of our in-depth telephone discussion, the CSC program had only one hired team member (the team leader); however, the program's expectation was that they would become operational by June 30, 2015.
Financing Strategy. The CSC program expects to bill third-party payers to ensure future sustainability of the program.
Partnerships and Collaborations. Since the Highlands CSB is located in a rural area with low population density, the CSB has cultivated strong partnerships and collaborations to build a robust referral network for client recruitment. The CSB deploys staff members in local schools, including day treatment programs in some of the high schools. The agency has had agreements for many years with a private college and a community college located within the agency's catchment area to address the behavioral health needs of their student bodies. For example, CSB staff regularly meets with the private university's counseling staff and is on hand for students in crisis. The CSB also partners with family advocacy groups.
Outreach to Potential Referral Sources. The CSB plans to leverage its partnerships and collaborations to recruit clients for its CSC program.
Western Tidewater CSB CSC Program
In addition to all CSC components, the Western Tidewater CSC program will include a 90-day independent training program, where clients experiencing difficulties with independent living will be provided with furnished apartments for 90 days as part of an intensive program to build skills for independent living.
Although the CSC program's primary target population is individuals with any early psychotic disorder, the program will also accept individuals without psychosis as long as they have ESMI.
At the time of our in-depth telephone discussion, the CSC program had enrolled six clients.
Partnerships and Collaborations. Western Tidewater CSB is located in a largely rural area and, as a result, has developed strong partnerships and collaborations to build a robust referral network for client recruitment. The CSB has partnerships with primary care providers and schools. Western Tidewater has long collaborated with court services units and law enforcement agencies; the agency employs two mental health court service liaisons and hosts a crisis intervention training program for law enforcement, dispatchers, etc., to teach them how to interact with mentally ill individuals.
Outreach to Potential Referral Sources. In addition to conducting internal outreach to look for referrals from within the CSB, the agency plans to leverage its partnerships and collaborations to recruit clients for its CSC program. The CSB recently reached out to the local Family Assessment Planning Team (see footnote on proximal referral sources in the section "Implementation Strategy as Designed by the ESMI Programs") to discuss possible referral sources.
As already mentioned, the only grantee in the state is JMHC, a large and established provider located in Madison in Dane County. JMHC enrolled its first client on March 6, 2015; at the time of our second discussion with program staff, JMHC was fully staffed although it was still recruiting a peer support staff, and the program had enrolled eight clients.
Partnerships and Collaborations. JMHC has a close relationship with the local community hospitals and had close prior relationships with the research staff of University of Wisconsin, Madison, and was able to recruit a researcher and a graduate student to assist with the evaluation component of the program.
Outreach to Potential Referral Sources. In addition to conducting internal outreach to look for referrals from within JMHC, the agency plans to tap into its extensive referral network and leverage its partnerships and collaborations to recruit clients for its CSC program. The agency will seek referrals from discharge staff at inpatient units, emergency departments and the JMHC Emergency Services Unit, primary care practices, local child and youth mental health programs, three large public high schools and alternative high schools, the University of Wisconsin-Madison, local health care providers, including the University of Wisconsin-Madison Medical Center, local shelters, the criminal justice system, and churches.
Program Strategy: Tier 3 States
The state awarded the set-aside contract to the Jefferson Center for Mental Health in June 2015, and by the time of our telephone discussion, the grantee had not yet begun to hire staff. The state anticipates that the CSC program will become operational by fall 2015.
The state awarded the set-aside contracts to two CMHCs in April 2015. As of June 2015, the programs were hiring and training staff but neither team had enrolled clients. Although the trainings are provided jointly to both programs, they differ in some respects--for example, only one of them is planning to offer supported employment services.
Partnerships and Collaborations. One of the grantees has a close relationship with a local substance abuse provider organization--the CSC program is leveraging the partnership to ensure access to substance abuse services for its future clients.
Outreach to Potential Referral Sources. One of the CSC programs is exploring the use of social media tools as a recruitment strategy. Also, at the time of our telephone discussion, one of the programs was planning a presentation to a local community group that includes hospitals, faith-based organizations, and youth groups among others, to develop a referral base.
The Children's Cabinet, a Reno-based provider, is the state's sole set-aside grantee and FEP program. At the time of our site visit, the program was not yet operational but extensive planning had been conducted over the previous year and hiring for staff was beginning. The program anticipates becoming operational later in 2015.
Partnerships and Collaborations. The Children's Cabinet has an extensive network of community partners, including local hospitals, a mobile crisis program for adolescents funded through the Department of Children and Family Services, primary care providers, the Washoe County School District and its health center, and family organizations. The program model integrates staff from the Children's Cabinet with state-employed clinicians.
Outreach to Potential Referral Sources. The Children's Cabinet plans to leverage its partnerships and collaborations to recruit clients for its FEP program. Particularly valuable outreach opportunities include suicide screenings for adolescents and meetings with primary care providers and families. The program will be able to reach a statewide audience at the Mental Health Summit held annually in Reno.
The Central Washington Comprehensive Mental Health, a large multispecialty human services provider, is the state's sole grantee; New Journeys, the ESMI program modeled after the EASA model and located in Yakima County, targets individuals with FEP. The program anticipates becoming fully operational by September 2015.
Partnerships and Collaborations. The agency has strong connections with multiple community partners, including law enforcement.
Outreach to Potential Referral Sources. During the initial phase of implementation, the program expects to seek referrals largely from within the agency, in particular from emergency departments, crisis intervention teams, and inpatient psychiatric services. Over time, depending on capacity, the program expects to seek referrals from a broad range of community partners.
Description of the States' Evaluation Strategy
To ensure that ESMI programs paid for by the set-aside funds implement interventions that yield the desired outcomes, the SAMHSA guidance to states includes language related to collecting data with the goal of "demonstrating program effectiveness."1 SAMHSA did not provide directives about how to conduct the program evaluation, and based on reports from states, additional guidance on the agency's "expectations for data collection and reporting" promised as part of the guidance did not follow. The guidance did highlight policymakers' interest in clinical and social outcomes, such as prevention of deterioration of cognitive functioning and fewer disruptions of employment. Given the states' discretion in strategy for using the set-aside funds to evaluate the ESMI programs as implemented, we explored whether the states have an identifiable evaluation strategy--designed by the states, grantees, or both--and whether separate set-aside funds have been carved out for evaluation. We also explored whether programs are leveraging academic partnerships and existing state information systems (see Table 4).
Here, we describe the states' plans for evaluating fidelity and outcomes, whether intermediate and typically measured as part of assessments of quality of care, or person-level outcomes capturing health status and well-being (see Table 4). Fidelity is commonly described with structure domains, intermediate outcomes are described with process domains, and person-level outcomes are described with outcome domains. We explored the states' approach for selecting evaluation domains targeted for the evaluation, data sources for the construction of measures, and the frequency of assessments. Because Tier 1 and Tier 2 states were farther along in their program implementation, we describe evaluation plans by tier. Our exploration of domains for the program evaluation component was designed in collaboration with NIMH.
Program fidelity is the degree to which the program is implemented "as intended," according to the known characteristics of the program model. Our examination of states' plans for evaluating fidelity included whether the strategy captured the following structural domains: staffing, caseload size, team orientation, off-hours availability, and outreach. Similarly, our examination of states' plans for evaluating intermediate outcomes and quality of ESMI care included whether the strategy captured the following process domains: time from referral to intake; identifying and addressing suicide risk and substance use; quality of engagement; adequacy of family involvement, metabolic testing, and pharmacological management; and frequency of medication management visits. Last, we determined if states sought to capture the following person-level outcomes domains: incidence of suicidal behavior, symptom stability, substance abuse behavior, schooling/employment, and criminal justice system involvement.
|TABLE 4. Evaluation Strategy by State|
|By Tier and State||State Prescriptive
|ESMI Programs Contacted||Fidelity Monitoring Plan
|Outcomes Monitoring Plan
(Process and Outcomes Domains)
|Other Aspects of
|California||No||Alameda||Plan not well defined yet||Process Domains:
Actively engaged in development of evaluation strategy with program developers (The Felton Institute)
|County record keeping on homelessness and employment, clinical records from FSP|
|Fresno's FOT||Plan not well defined yet||Process Domains:
|Not specified||EHRs Primary data collection, Medicaid|
|Imperial||Plan not well defined yet||Outcomes Domains:
Using evaluation methods from PIER program. Hired consultant on evaluation.
|EHR (AVATAR) Primary data collection|
|San Luis Obispo||Plan not well defined yet||Outcomes Domains:
Working with CalPoly to develop evaluation plan
|Data source(s) not well specified|
||IOL program is not monitoring process or outcomes of care||Not specified||Data source(s) not well specified|
Process and outcomes domains are routinely assessed; exact frequency was not specified.
|Data source(s) not well specified|
|New York||Yes||Northern Rivers Family Services||Fidelity Domains:
Process and outcomes domains are routinely assessed; exact frequency was not specified.
|Idaho||Yes||St. Luke's Regional Health Center||Plan is in development||Process Domains:
PHQ, AUDIT, and functioning scales
|St. Luke's EHR|
|North Carolina||No||OASIS (including OASIS Carrboro and Wake STEP)||Fidelity Domains:
EP-TAP Brief Assessment of Cognitionb
|NC Tracksc (Medicaid information system)
Alpha (internal data system)
EPIC (hospital-based EHR)
|Texas||Yes||ePEP (Dallas Metrocare)||Fidelity Domains:
CANS, ANSA, PANSS, MASQ-D30, and Birchwood SFS
|MHMRA of Harris County||Fidelity Domains:
CANS, ANSA, BPRS, PANSS, Columbia suicide rating scale.
|Virginia||No||Alexandria CSB||Fidelity Domains:
As determined by the work group, all CSBs will use the modified Colorado index and a self-reporting tool that includes components of the Mental Health Consumer Satisfaction Youth Satisfaction Survey and the Illness Management and Recovery Self-Report Scale
|NOMSc and TEDSc|
|Fairfax-Falls Church CSB||Plan is in development with assistance from an outside evaluator|
|Highlands Community Services||Fidelity Domains:
|Western Tidewater CSB||Plan not well defined yet|
Reach and Recovery scale, PANSS
|JMHC's EHR (Omnis)d|
|Colorado||N/A||Jefferson Center for Mental Health||Plan not well defined yet||Plan not well defined yet||Not specified||Data source(s) not well specified|
|Iowa||N/A||CMHC, Des Moines||Plan not well defined yet||Plan not well defined yet||Not specified||Data source(s) not well specified|
|CMHC, Cedar Rapids||Plan not well defined yet||Plan not well defined yet||Not specified||Data source(s) not well specified|
|Nevada||N/A||The Children's Cabinet of Northern Nevada||Plan not well defined yet||Plan not well defined yet||Partners/Consultants:
Developing evaluation plan with Oregon (EASA) consultants.
|Data source(s) not well specified|
|Washington||N/A||Central Washington Comprehensive Mental Health||Fidelity Domains:
Working with researchers at the University of Washington to develop evaluation plan.
|Data source(s) not well specified|
States' Evaluation Plans: Tier 1 States
Consistent with the highly decentralized nature of the state, the SMHA was not prescriptive about the evaluation component of the implementation of the set-aside policy. Each of the California counties with which we held discussions included an evaluation component, but they were at different development stages. Alameda and San Luis Obispo are working with academic partners to develop and implement evaluation methods (The Felton Institute and Cal Poly, respectively), and Fresno and Imperial are developing evaluation plans with input from consultants.
Fidelity. Although all counties have access to information on several relevant structural domains, they did not have well-defined plans for fidelity monitoring.
Process and Outcomes of Care. Alameda County plans to evaluate quality of delivery of clinical services, but respondents did not provide details. Fresno County plans to evaluate hospitalization and use of crisis services. No other county reported active efforts to assess process domains. All counties are planning to assess person-level outcomes. Alameda County plans to assess living situation and employment status at intake and annually until discharge--however, the county is considering more-frequent assessments. Alameda County uses clinical records from the FSP as a data source to evaluate outcomes. Fresno tracks employment status. Imperial County collects data on criminal justice involvement, employment, and housing status. Fresno and Imperial counties primarily use EHR data (AVATAR,ab a new system in the case of Imperial County) supplemented with primary data collection and Medicaid data sources to evaluate outcomes. San Luis Obispo County routinely collects data on educational outcomes such as attrition rates and academic success.
The SMHA was not prescriptive about the evaluation component of the implementation of the set-aside policy in the state. While the IOL program did not have a well-defined evaluation strategy, the STEP Program has been involved in active evaluation projects designed to document the effectiveness of the model for many years. The IOL program does not have academic partners but the program collaborates with Hartford Hospital's research center. The STEP program has leveraged its longstanding partnership with Yale University as a collaborative program of the University's Department of Psychiatry.
Fidelity. Both programs collect data on clinician skill and comfort with the model as their main targets for monitoring program fidelity.
Process and Outcomes of Care. The IOL program does not monitor process or outcomes at this point. The STEP program considers the assessment of process of care to be central to the program's evaluation efforts. Respondents reported that they evaluate engagement with the program but did not provide details on domains, data sources, or frequency of assessments. However, they are interested in examining demographic disparities in these intermediate outcomes. The program routinely collects data on a variety of person-level outcomes domains, including severity of psychotic symptoms, DUP, hospitalizations, vocational outcomes, and quality of life. The program also tracks cardiovascular outcomes, social functioning and recovery, and community tenure (i.e., community dwelling, institutionalized, etc.).
The SMHA was highly prescriptive in its evaluation strategy, which included an allocation of set-aside funds for the conduct of the evaluation component. Like other aspects of the policy implementation, the evaluation component has benefited from the close collaboration between the SMHA and Dr. Lisa Dixon's team at the Center for Practice Innovations. The newly developed OnTrackNY Program at the Northern Rivers site is working with Dixon's team to develop an evaluation plan.
Fidelity. OnTrackNY programs tap into routinely collected data to assess all the fidelity domains mentioned above. For example, OnTrackNY routinely assesses whether teams are appropriately staffed and have off-hours availability, and whether the staff regularly meets as a team.
Process and Outcomes of Care. OnTrackNY programs evaluate process of care through measures that capture all the process domains mentioned above. For example, OnTrackNY assesses whether substance use is identified and addressed, whether the team meets with family members, and the adequacy of antipsychotic regimens. OnTrackNY also assesses the outcomes domains mentioned above, including symptom stability and education/employment status, and also assesses quality of life.
States' Evaluation Plans: Tier 2 States
The SMHA was moderately prescriptive in its evaluation strategy. As reported by respondents, the state instructed St. Luke's Regional Health Center, its sole grantee, to use a specified amount of the set-aside funds to conduct data analysis and produce outcomes reports. The state required that demographic and diagnostic data be collected at initiation of services and that selected outcomes information be collected upon discharge. The design of the evaluation component has been conducted through a collaborative iterative process between the SMHA and the grantee. St. Luke's does not have academic partners at this point. The grantee is not planning to leverage existing state information systems for the evaluation component.
Evaluation of Fidelity. The state is working with St. Luke's to develop a plan to monitor fidelity.
Process and Outcomes of Care. The state and the grantee are working on the development of a plan to evaluate process and outcomes. The outcomes evaluation initiative would leverage newly collected data (i.e., demographic, diagnostic, and selected person-level outcomes) and possibly data routinely collected by St. Luke's. Currently, St. Luke's collects information pertaining to symptom severity, problem drinking, hospital readmission rates, emergency department visits, and interactions with law enforcement. These outcomes are assessed with standardized scales (e.g., the Patient Health Questionnaire [PHQ]) and functioning scales; the latter are administered every 90 days. The state is also interested in evaluating participant satisfaction.
The SMHA allocated a portion of the set-aside funds to conduct the evaluation component, and the agency identified and recommended several data elements that applicants should consider in the evaluation design, but the agency was not prescriptive in its evaluation strategy. The original OASIS Program at the Carrboro in Orange County was given additional funding to develop a technical assistance and evaluation component for the three North Carolina sites, which they named Early Psychosis Technical Assistance Program (EP-TAP). Key components of OASIS' EP-TAP include maintaining a quality assurance database, routine outcomes monitoring, and feedback-informed treatment protocols. OASIS has had a long-term operational and academic partnership with UNC. OASIS' evaluation strategy does leverage state information systems as it utilizes a newly developed state Medicaid information system (NC Tracks) and an outcomes performance system (TOPPS).
Evaluation of Fidelity. The EP-TAP monitors program fidelity by assessing staff-to-patient ratios, quality and frequency of individualized treatment planning and multidisciplinary team meetings, and other domains.
Process and Outcomes of Care. Process and outcomes of care are evaluated through the EP-TAP. Process domains include weekly multidisciplinary team collaboration, integrated substance abuse counseling, routine assessment of suicide risk, and evidence-based pharmacological care. Outcomes domains include housing and employment status and symptom severity assessed through standardized scales such as the Brief Assessment of Cognition. These data are then entered into NC Tracks. Sources of data for the outcomes evaluation component are National Outcomes Measurement System (NOMS), North Carolina Treatment Outcome Program Performance System (TOPPS), Alpha (internal data system), and EPIC (hospital-based EHR). Client-level data are collected at baseline and every six months to inform program-level planning and fidelity scaling. Client-reported and clinician-reported data are used on a regular basis to guide day-to-day interventions; other data including results of standardized assessments are used quarterly to inform individualized treatment planning.
The SMHA was moderately prescriptive in its evaluation strategy, which included the allocation of a portion of the set-aside funds to conduct the evaluation component and the requirement that programs collect and report fidelity-relevant information described in the CSC curriculum. However, the state did not prescribe the evaluation strategy and each program was free to develop a suitable plan. Academic partners of each of Texas' two grantees, MHMRA of Harris County and ePEP in Dallas, designed the components for the evaluation plan. While MHMRA of Harris County contracted with the UT Medical School at Houston to conduct the program evaluation, the Dallas-based ePEP contracted with the UT Southwestern Medical Center to conduct the program evaluation. The evaluation teams in both MHMRA of Harris County and ePEP program in Dallas are planning to leverage information collected by the CMBHS, the state's electronic web-based data system.
Evaluation of Fidelity. While the MMHRA of Harris County did not have a well-specified strategy for assessing fidelity, the ePEP program had a clear strategy as documented in the Metrocare Intervention plan. The ePEP program's approach for assessing fidelity to CSC's structure of care captures all the fidelity domains mentioned above, including monitoring the adequacy of staffing and caseload. Fidelity data will be collected through client and clinician interviews.
Process and Outcomes of Care. Neither program discussed plans for assessing intermediate outcomes with process domains. All publicly funded providers in Texas perform the Child and Adolescent Needs and Strengths Assessment (CANS) and Adult Needs and Strengths Assessment (ANSA) at intake, every 90 days, and at discharge. These scales collect information on a number of outcomes domains (e.g., symptom domains, risk behaviors, social functioning, psychiatric hospitalizations).ac Both evaluation teams are considering the use of these assessment tools in addition to other outcomes measures. Outcomes information for MHMRA of Harris County will be collected using the following scales: Brief Psychiatric Rating Scale (BPRS), Positive and Negative Symptom Scale (PANSS), and the Columbia suicide rating scale. The BPRS and PANSS will be administered during program intake, after three months of services, and after one year of services. The evaluation team will also collect data on a control group of patients who meet the eligibility criteria but are not participating in the CSC program and are receiving usual care at the clinics affiliated with MHMRA of Harris County. The evaluation team will train MHMRA staff to administer these scales and collect data to ensure sustainability of the evaluation component; the evaluation team will analyze collected data and report back to the state.
The annual baseline assessment for ePEP in Dallas will collect information related to "...residential status, mental health service utilization, arrests and incarcerations, and employment/disability status" during the past year of treatment. Clinical records will be reviewed to collect information on "...number of clinic visits for all levels and types of care, number of appointments missed, number of emergency care visits and hospitalizations." The research team plans to compare up to 60 clients enrolled in the ePEP program to 60 control subjects who meet the eligibility criteria but are not participating in the CSC program and are receiving usual care at the Metrocare clinics. Individuals in the ePEP group and the control group will receive a baseline assessment, interval assessments every few months, and a final outcomes assessment at the end of the year. Assessments to collect outcome measures will be conducted by a study coordinator. Outcomes scales include: the PANSS, Mood and Anxiety Symptoms Questionnaire (MASQ-D30), and the Birchwood Social Functioning Scale (SFS). Participants will also complete sections of the baseline assessment at the end of a year. Clinical records will also be reviewed.
Although the request for proposals issued by the SMHA to selected grantees stated that the agency would require "quarterly data reporting and outcome...with required data elements to be determined once SAMHSA guidelines for federal data reporting expectations are released," the SMHA was not prescriptive in its evaluation strategy and no distinct funds were allocated specifically for program evaluation. However, the strategy used to assess program effectiveness is evolving through a collaborative process between a state work groupad and the eight grantees. This initiative had just begun by the time of our site visit, a delay the state attributed to their expectation that they should wait until the promised federal guidance on the evaluation component was issued. At this time, none of Virginia's CSBs have partnered with an academic institution for program development, implementation, or evaluative purposes. The grantees are planning to leverage information collected by the state systems through EHRs, however, not all CSBs in the state use the same EHR.
Evaluation of Fidelity. Supervisory notes and team meeting logs will track fidelity to the recovery orientation and team approach for the Alexandria Program. The EHR will be used to monitor that all CSC service elements are delivered according to client needs and preferences. Adherence to the model will be monitored by the team supervisor on an ongoing basis. The Fairfax CSB has budgeted for an outside evaluator to assist CSC program staff in developing data collection tools and fidelity models to ensure adherence to the RAISE model. Highlands Program plans to ensure fidelity through weekly team meetings, quarterly meetings with the Advisory Committee, and regular technical and administrative assistance calls with the NAVIGATE team. Highlands Community Service contracted with an outside consulting company, TurnKey KB & Associates, to assist with monitoring program fidelity and to serve as a liaison between Highlands Community Service, the SMHA, SAMHSA, and FEP/CSC experts.
Process and Outcomes of Care. The state does not have a clear strategy for collecting process domains. The SMHA's evaluation work group discussed possible outcomes domains and data collection strategy with the Center for Practice and Innovation and the eight set-aside providers in the state. The work group eventually decided on the following domains: hospitalizations/engagement with crisis or emergency services, criminal justice involvement, symptomatology, family engagement, satisfaction with services (self-report), measure of recovery (self-report), and engagement in employment or school-related activities. The work group decided to use a modified Colorado symptom index to collect measures related to symptomology. The state will also utilize data collected for NOMS and Treatment Episode Data Set (TEDS) reporting, which include hospitalization, employment and school engagement, and emergency department visits. For the self-report measures, a member of the data and evaluation group at the SMHA developed an instrument that includes components of the Mental Health Consumer Satisfaction Youth Satisfaction Survey and the Illness Management and Recovery Self-Report scale. The SMHA will collect the selected outcome measures from the CSBs on a quarterly basis and update information in a state-level data repository system. Currently, the CSBs are modifying their EHR systems to be able to collect the selected outcomes. Although the state did not specifically plan to collect measures related to suicidality and substance use behaviors, respondents mentioned that CSBs would be able to collect such data easily.
Wisconsin was not prescriptive in its evaluation strategy, and no distinct funds were allocated specifically for program evaluation. Program fidelity will be overseen by the SMHA's contract manager, but JMHC, the sole grantee, developed an evaluation plan overseen by the agency's Continuous Quality Improvement Committee. JMHC has leveraged a valuable partnership with the University of Wisconsin, Madison, to pursue the evaluation. As previously mentioned, the outcomes evaluation component will be designed and overseen by a researcher affiliated with the university free of charge. Moreover, university-affiliated researchers provided JMHC with the resources needed to train the CSC staff on the administration of structured clinical interviews during the intake process. Data collected by JMHC is uploaded to the state's centralized web-based data system on a monthly basis.
Evaluation of Fidelity. To measure adequate staffing and outreach, JMHC will monitor the number of outreach and engagement visits completed monthly or quarterly and the number of program participants seen by CSC providers. Monthly meetings will be held to monitor program fidelity through a review of supervision logs and a review of individual/team trainings.
Process and Outcomes of Care. JMHC does not have a well-specified plan to assess process of care. However, the grantee has a clear strategy for evaluating outcomes domains. JMHC will tap into its EHR system, Omnis, to collect outcomes information including employment, hospital days, jail days, housing status, suicidality, substance use, and whether people are meeting self-set goals. Data is uploaded to the SMHA's centralized web-based data system on a monthly basis. The clinic is building a new EHR with additional fields to be able to capture more outcome domains for the FEP program. Measures are collected during clients' one-on-one visits with clinical staff, and progress of clients is reviewed every week. The prescriber will also collect measures related to the clinical symptoms of psychosis using the Reach and Recovery scale and PANSS every six months.
States' Evaluation Plans: Tier 3 States
Colorado, Iowa, Nevada, Washington
None of the Tier 3 states with which we held discussions were prescriptive with their program evaluation strategies. Nevada was the only Tier 3 state that specifically dedicated set-aside funds to the conduct of a program evaluation. The Washington SMHA is working with researchers at the University of Washington to develop a research and evaluation plan for the New Journeys Program based in Yakima County. Tier 3 states were unable to report whether their ESMI programs would leverage existing state information systems, as many states were still in the very early stages of developing evaluation plans.
Only Washington reported having a tentative plan for evaluating fidelity (adopting fidelity measurement methods used in Oregon's EASA model) and outcomes of care (monitoring hospitalizations, employment, and educational attainment); none had plans for evaluating process of care.
Barriers and Facilitators to Implementation
Population Density. States face the challenge of matching programs with the appropriate populations, particularly in non-urban areas. Some states, such as Nevada, have avoided this issue for the moment and have established their initial ESMI programs in urban areas. However, this leaves their non-urban areas uncovered. Some states, such has Oregon, have decided to take on the challenge of establishing an ESMI program in a non-urban region, which requires referral and provider networks over a much larger area. Over time, these programs will face challenges in providing services to clients who must travel long distances unless they can develop alternative methods for providing services remotely. Future programs are likely to benefit from lessons learned in these early attempts to deliver services to non-urban populations with FEP and other ESMI.
Narrow Eligibility Criteria. FEP programs such as CSC have, by design, more narrowly defined eligibility criteria than most social service programs. While the reasons for focusing on non-affective psychosis are clear, this does create challenges for CSC programs--only one of which is identifying a large enough group of clients to make the program cost-effective. Since many of the organizations that refer to CSC programs do not have the clinical expertise to determine eligibility and because they are often searching for resources for the young adults they serve, a large portion of referrals are likely to fail to meet criteria for the CSC program. In many cases, programs will not simply be able to turn these clients away, but will want to refer them to other appropriate programs in the community. In addition, refusing to accept clients referred from community agencies may lead to misunderstandings among agencies. New programs may benefit from lessons that more-established programs have learned regarding managing the intake and eligibility screening process.
Hiring Staff. Workforce shortages are common in public mental health service systems across the country. Shortages are particularly acute in rural areas, but they also affect urban areas where clinicians have options that pay higher salaries than public sector psychiatry. These shortages may affect CSC and similar programs more acutely because of their need for clinicians to provide specialized therapy. CSC often has the option of using existing staff members who have related skills and can devote a portion of their time to the program, or hiring new staff who will be specialized in the CSC program. In many cases, the strategy of leveraging existing staff will be a more successful strategy because of the difficulty in hiring new staff members. This scenario is best illustrated by the Texas experience in which the CSC program that hired new staff took far longer to become operational relative to the program that hired within the parent agency.
Evaluation Capability and Focus. Grantees may have very limited data collection and analysis capability, which will undermine efforts to systematically evaluate the effectiveness of ESMI programs. Moreover, although SAMHSA had indicated to states that guidance on evaluation would be forthcoming, states have not received much of it to date. As a consequence, states have developed evaluation plans that vary widely in their scope and breadth. The lack of guidance clearly contributed to delays in the implementation of ESMI programs in Virginia, and may also have been a barrier in other states. Although all the states in this study were developing evaluation plans for their ESMI programs, they all faced considerable challenges in this area.
Strong State Guidance. Grantees in several states reported that they very much appreciated and valued state guidance and logistical assistance facilitated by the SMHA in connection with implementation and evaluation activities.
Existing Programs for Transitional Age Youth. States took advantage of existing programs serving at-risk TAY, some of which have connections to mental health treatment facilities, as well as a broad range of social support services. Although the TAY programs have a much broader target population than CSC and other ESMI programs, linking an ESMI program to a TAY program can facilitate implementation. The TAY programs may come into contact with young adults experiencing a FEP through their interaction with youth serving clinical and social services agencies. Second, TAY programs may include staff with skills that are valuable to ESMI programs, including expertise in care coordination. Several CSC programs in our case study states either help to cover the salaries of staff associated with existing TAY programs and/or provide specialized training in the identification of FEP to TAY program staff.
Existing Provider Networks. States have found it useful to draw on existing community-based networks of clinical and social service providers in establishing their ESMI programs. Most commonly, ESMI programs are situated within large social service agencies, which not only have a history of serving children or youth, but also have extensive connections to specialty mental health providers, schools, and other potential sources of services and referrals within their region. States may also take advantage of organized networks or coalitions of provider organizations. These networks can provide referrals to ESMI programs because they are likely to come into contact with young adults experiencing a first episode of psychosis or serious non-psychotic illness and have established lines of communication.
Existing Centers of Expertise in CSC and Other ESMI Models. Several states that had no previous experience with early intervention programs relied heavily on advice from consultants from established centers with expertise (e.g., Center for Practice Innovations at the New York State Psychiatric Institute, the Felton Institute in San Francisco). These centers provided input on a wide range of issues including program location, staff composition, referral networks, clinical skills, and evaluation methods.