Feasibility Study for Demonstration of Supported Education to Promote Educational Attainment and Employment among Individuals with Serious Mental Illness: Final Report. 3. Literature Review

09/01/2015

3.1. Introduction

The objective of this literature review is to complement and expand on prior reviews of the literature published around SEd interventions. Two systematic reviews of SEd approaches have been published relatively recently (Leonard & Bruer, 2007; Rogers, Kash-MacDonald, Bruker, & Maru, 2010) and several other articles summarize the state of SEd program implementation and research (e.g., Chandler, 2008; Ellison et al., 2013; Manthey et al., 2014; Mueser & Cook, 2012; Parrish, 2009; Unger, 2011). To compliment this prior work, the current literature review adds studies published from 2010 through 2014. Within these recent studies, we have placed particular emphasis on publications that include outcome-oriented trials. Second, this review includes publications before 2010 that were excluded by Rogers & colleagues (2010) and Leonard & Bruer (2007). These publications include studies that describe SEd program models (without reporting on program outcomes), process and implementation evaluations, and publications that summarize qualitative results exclusively. These publications are reviewed to help offer a description of SEd programs that are both currently being (and have historically been) used in the field. Finally, this review offers a slightly expanded definition of SEd interventions to include those that are education-focused without explicitly being referred to as "SEd."

More specifically, this report draws on the existing published literature to:

  • Describe the characteristics of SEd interventions (service characteristics, populations served, financing strategies, and implementation challenges).

  • Report on the impact of SEd interventions.

  • Identify gaps in the published literature about SEd interventions, particularly those relevant to the feasibility and design of future demonstration activities.

3.2. Literature Review Method

This literature review was guided by the definition of SEd adopted by Collins & Mowbray (2005) in their survey of SEd programs--"a specific type of intervention that provides support and other assistance for persons with psychiatric disabilities for access, enrollment, retention, and success in post-secondary education."

Search terms for the preliminary literature search included SEd or supportive education, education OR school OR post-secondary education and (treatment or intervention), and employment and (treatment or intervention). All of these search terms were paired with mental illness, mental disorder, SMI, or psychiatric disability/disabilities. Search terms such as "education" and "employment" were also used to broaden the literature reviewed to potentially include interventions focused on post-secondary education support and intervention that may not have been labeled explicitly as "supported education." Search engines used included PubMed, the Web of Science (includes Science Citation Index Expanded and Social Sciences Citation Index), PsycINFO, and the Education Resources Information Center. Both peer-reviewed publications and gray literature (e.g., government or university-published reports) were included in the literature review. The search was limited to articles written in English and published from 1990 to November 2014. Articles published outside of the United States were included. In addition to these keyword searches, we examined citations contained in each article and citations from key SEd review articles to identify other potential articles to include in the review. Please note that we did not include unpublished articles in this review. Unpublished work, conference proceedings, or manuscripts in press will be reviewed within a subsequent report resulting from our environmental scan.

This keyword search and supplemental article review identified 150 abstracts for consideration. In reviewing these abstracts, we excluded 75 publications, because the study:

  • Focused exclusively on SE or employment without an education component.

  • Examined a traditional occupational therapy intervention, "wellness" education programs, or other psycho-educational programs (e.g., programs designed to help manage symptoms).

  • Included only children.

  • Included only a single case example (i.e., one consumer's story).

  • Was only theoretical, without any emphasis on SEd program or evaluation data.

Acceptable SEd studies included descriptive program model summaries, original research (both outcome and process evaluations), and review articles. For original research publications, we included pre/post evaluations, correlational studies, experimental studies, and quasi-experimental studies.

After applying the exclusionary criteria, 75 publications were left for consideration. We did not carefully re-analyze the 13 SEd outcome studies published from 1989 to 2009 and included in the systematic review conducted by Rogers & colleagues (2010). Instead, a synthesis of the Rogers review is included within this report, along with a summary of some seminal studies. We did, however, review several articles published from 1989 to 2009 that were not included in the Rogers review, likely because of their focus on program model descriptions, process evaluations, or qualitative research. We also identified 31 articles that had been published on SEd since 2010; 16 of these were original research studies designed to examine the impact of a SEd intervention. We have placed particular emphasis on these 16 studies, not all of which included equally rigorous designs and study methods. Some of these publications describe preparations for a SEd trial or characteristics of participants currently involved in an ongoing trial. The strengths and weaknesses of these recent SEd studies will be discussed in this chapter.

3.3. Characteristics of Supported Education Programs and Their Participants

Several publications on SEd summarize the characteristics of the programs themselves and their participants. This report section describes the populations typically served by SEd programs; traditional SEd models, settings, services and staffing; financing; efforts to integrate SEd approaches with SE and related programs. In 2011, the U.S. Department of Health and Human Services (HHS) Substance Abuse and Mental Health Services Administration (SAMHSA) published a toolkit for SEd programs to structure and guide SEd program implementation. Appendix B of this toolkit provides a Supported Education Fidelity Scale developed by researchers at the University of Kansas, as well as a scoresheet for programs to examine adherence to key aspects of the SEd program model.

3.3.1. Participants in Supported Education Programs

Eligibility criteria for individuals served in the SEd programs described in the published literature vary slightly across programs. All programs require participants to have some history of psychiatric disability without any age restriction. Some programs go further to require specific a specific duration period (e.g., "for 12 months") for the mental illness while others target individuals experiencing a first-episode of mental illness or psychosis. Several programs describe that program participants were required to have an interest in pursuing post-secondary education, basic English fluency, and a willingness to utilize mental health services. Some programs also require that participants be actively enrolled in mental health treatment, even sometimes requiring adherence to a medication regimen (e.g., Gutman, Kerner, Zombek, Dulek, & Ramsey, 2009). All programs had some prior education eligibility criterion, but this criterion differed slightly across programs. Some programs required participants to have a high school diploma or General Educational Development (GED) (or to least have them near completion; e.g., Collins et al., 1998), whereas other programs did not have this requirement and described active work with participants to acquire GEDs (e.g., Hain & Gioia, 2004). Some programs explicitly stated that participants needed to show no evidence of a significant drug or alcohol problem (Gutman, 2008), no pre-morbid history of mental retardation or neurological disorder (e.g., Nuechterlein et al., 2008a). Another program excluded individuals with unstable housing or homelessness and those lacking a support system (e.g., Hutchinson, Anthony, Massaro, & Rogers, 2007). Another excluded individuals with a history of violence (Holter & Paul, 2004).

Looking across SEd programs that were operating at that time, Mowbray & colleagues (1996) noted that SEd participants tended to be younger, more educated, and higher functioning than individuals with SMI from more general non-SEd program samples. For example, many participants in the Michigan-Supported Education Research Project (MSERP) had significant problems with mental health symptoms, social skills deficits, and histories of substance abuse; however, these issues did not prohibit participants from being able to stay involved in the SEd program (Collins et al., 1998). Unfortunately, the SEd program outcome literature is too premature to conclude which types of individuals are best positioned to benefit from SEd approaches.

Some recent SEd approaches have adapted and tailored SEd programs to better fit special populations. For example, Shor & Aivhod (2011) describe the rehabilitation beit midrash adaptation of a SEd program that maintains the principles and practices of psychiatric rehabilitation while implementing the approach in a culturally oriented context. All program participants were men, 70% of whom lived in rehabilitative residential facilities and were Orthodox or strictly Orthodox Jews. This descriptive article discusses using Judaic program content and values as a method to advance the rehabilitation process and enhance program participants' sense of belonging and inclusion. As another example, Smith-Osborne (Smith-Osborne, 2012a, 2012b) describes the design, development, and adaptation of a SEd program specifically for veterans. Adaptations were made based on a participatory action research approach that worked to engage stakeholders in the community, U.S. Department of Veterans Affairs (VA), and higher education settings. Program components are modified to reflect the veteran student context. For example, veterans share a house (including students and nonstudents), rather than participate in a more traditional rehabilitation housing program. Budgeting includes VA disability pension instead of Supplemental Security Income (SSI) benefits. This program's impact is currently being tested in a randomized controlled trial (RCT).

3.3.2. Supported Education Program Models

There are several different approaches to SEd, each designed to help individuals with SMI succeed in the post-secondary education environment. These approaches vary according to their setting location, service array mix, and integration with the mainstream post-secondary education environment.

Since the early 1980s, post-secondary institutions and mental health providers have developed SEd programs. Historically, some of these models have been "owned" and developed via leadership within the college system, whereas others have their origins and leadership from the mental health specialty system. The earliest SEd models were classroom-based (Walsh, Sharac, Danley, & Unger, 1991); however, with federal grant funding, SEd models were expanded from 1989 to 1994 to be implemented in a variety of settings (e.g., hospitals, mental health agencies, clubhouses) (Unger, 1998). On-site and mobile support models have now been added to these traditional, self-contained, classroom-based models. Federal grant funding also promoted the use of clubhouses across the United States to disseminate SEd via a free-standing organization (separate from the education or mental health systems).

In one of the first classifications of SEd programs, Unger (1990) characterized three different types of SEd program models:

  • Self-Contained Classroom Model: Students with psychiatric disabilities attend closed, self-contained SEd classes on-campus (but separate from mainstream post-secondary classes). Classes typically use a structured curriculum and are time limited. Students are not initially integrated into regular classes, but they may participate in the activities and use the institution's resources. However, support is available from program staff for students as they progress and move into regular classes. Education specialists may be from the sponsoring program or the academic institution.

  • On-Site Model: These models are sponsored by a college or university at which SEd services are provided in an individual rather than a group setting. Students attend mainstream post-secondary education classes. Support services are typically made available to all students with disabilities and are enhanced by adding specialized mental health staff or a peer support group. The education specialist works exclusively at one site and typically has an office on the campus or program site from which he or she provides support services (see description of this position in Ellison et al., 2014).

  • Mobile Support Model: Students attend mainstream post-secondary education classes of their choice, but SEd services are provided by an agency (typically a mental health agency) external to the education facility. The SEd education specialist office is at the mental health agency. SEd program staff provide support, assistance, and problem-solving in an individualized, flexible way wherever this support is needed. The SEd education specialist travels to meet the student in the mental health agency, community, or campus or education program site.

SEd program models continue to grow and expand over time. Consequently, these historical, individual classifications have become less and less useful. SEd programs are becoming more eclectic as discovered by Mowbray, Megivern, & Holter (2003b) in their survey of SEd programs being implemented across the United States. This survey found no SEd programs that operated with only a classroom model. Meanwhile, the majority (66%) of programs were offered through clubhouses. The clubhouse model is typically a support program designed for people with serious and persistent mental illnesses. Participants are considered "members" (as opposed to "patients" or "clients"), and activities are recovery oriented and strengths based. Because of the number of clubhouse-based SEd programs, Mowbray & colleagues (2003b) added some other classifications of SEd program models to those originally developed by Unger (1990):

  • Clubhouse Full Model: These SEd programs are located at clubhouses and offer individual counseling (either by staff or peers). The full clubhouse model provided 0.5 full-time equivalent (FTE) or greater staff devoted exclusively to post-secondary education (excluding GED services), an educational unit in the clubhouse, and at least two services beyond individual counseling (e.g., mentors/tutors, educational software programs, group support, education liaisons, transportation services, recruitment/outreach). These services could be mobile.

  • Clubhouse Partial Model: These SEd programs are located at clubhouses that focus on post-secondary education with fewer services than the full model (e.g., less than a 0.5 FTE staff person, only one service offered beyond individual counseling).

  • Free-Standing Model: This model provides some component of its services on a college campus or provides mobile services but also includes services off-site at a central office. Free-standing programs offer two services beyond individual counseling with 0.5 FTE or greater staff focused on post-secondary education.

Some of the diversity represented by SEd programs described in the literature can be seen in Table 3-1. This is not an exhaustive list but offers a few examples of SEd program models discussed in the literature.

As shown in Table 3-1, some older models described in the literature strictly follow a traditional classroom-based model. For example, the Redirection Through Education (RTE) program established in 1973 in Toronto, Canada, offers self-contained for-credit and noncredit classes taught by program-hired faculty. Course completion leads to a program-specific graduation certificate. Meanwhile, other programs mix model approaches. However, these self-contained models are now rare in the United States. Consistent with Mowbray and colleagues (2003a), several SEd programs now integrate various model aspects into their program approaches (e.g., on-site and classroom-based). A few examples of integrated approaches include Laurel House, the Bridge Program, and Supported Education Enhancing Rehabilitation (SEER). Laurel House (http://www.laurelhouse.net/) is a clubhouse program written about in the late 1990s that offered social, vocational, and residential services to people with a history of psychiatric hospitalization. This model includes a mixture of the free-standing model (classes and support services were located in the clubhouse) but also included aspects of on-site support (service supports were also provided on-campus) (Dougherty et al., 1996). A more recently established program, the Bridge Program, offers 12 modules of self-contained classes on site at Columbia University. Students were then offered 6 weeks of on-site mentoring and support at Columbia University from occupational therapists to facilitate their integration into mainstream education courses or subsequent employment (Gutman, 2008). Meanwhile, the SEER program operated out of Spokane, Washington, described offering on-site classes at a community college, along with mobile support that follows enrolled students wherever they choose to pursue their education or employment goals (across the entire country, not tied to a specific post-secondary institution) (Hain & Gioia, 2004).

TABLE 3-1. Examples of SEd Program Models, Names, and Supporting Citations
SEd Program Model Model Description Sample Program Name,
Setting, and Citation
Classroom model Students attend closed, self-contained SEd classes on-campus (but separate from mainstream post-secondary classes).
  • RTE (Gilbert, Heximer, Jaxon, & Bellamy, 2004; Kidd et al., 2014)
On-site Students attend mainstream post-secondary education classes sponsored by a college or university where SEd services are provided in an individual (not group) setting.
  • Houston Community College System (Housel & Hickey, 1993)
  • California Community Colleges System (Jacobs & Glater, 1993)
  • Mott Community College (Unger, 1990)
  • Bridge Program, Columbia University (Gutman, 2008)
Mobile support model Students attend mainstream post-secondary education classes, but SEd services are provided by an agency (typically a mental health agency) external to the education facility.
  • South Beach Psychiatric Center in New York (Lieberman, Goldberg, & Jed, 1993)
  • Thresholds Community Scholars Program in Chicago (Cook & Solomon, 1993)
  • Spruce Mountain Inn (Unger, 1990)
On-site and mobile support model A combination of the on-site and mobile support models.
  • SEER community college program in Spokane, Washington (Hain & Gioia, 2004)
Free-standing model Provides several services off-site at a central free-standing office.
  • Unnamed mental health clinic-based program in Quebec, Canada (Beguet, Fortier, & Gauvin, 2004)
Free-standing and on-site Provides some service components on a college campus or provides mobile services but also includes services off-site at a central free-standing office.
  • Laurel House (clubhouse model) (Dougherty et al., 1996)
  • On-campus services with county mental health agency support (Thompson, 2013)

3.3.3. Supported Education Program Settings

The behavioral health care system for individuals with SMI is complex and involves multiple sectors. Service sectors that provide support for educational and employment outcomes include specialty mental health, primary and post-secondary education, vocational rehabilitation (VR), and the Veterans Health Administration (VHA) service systems. Examples of SEd program approaches are used in all of these settings.

Mowbray & colleagues (2003a) conducted a national survey of known SEd programs across the United States. This survey found that the majority of SEd programs (66%) were offered through clubhouses; these clubhouse programs were extremely varied in terms of the amount and diversity of service approaches. Other than clubhouse-based approaches, the next most common setting was within a post-secondary institution. Mowbray & colleagues (2003a) did note a handful of SEd programs that were not located at either a clubhouse or university-based site (i.e., mental health agency/provider).

Because many SEd programs are often directly affiliated with a community college or university setting, Collins & Mowbray (2005) conducted another national survey. This time, they surveyed campus disability service directors and queried these post-secondary schools about the presence of SEd programs. According to the survey, most of the campus-affiliated SEd programs were located off campus (72%). Most of these SEd programs were managed by a mental health agency (68%), but some were operated by a clubhouse or vocational program (12%) or college or university (19%), or they were located in another setting (24%). The majority of these campus-affiliated programs focused on both post-secondary school enrollment and retention (58%), as opposed to solely enrollment (16%) or retention (26%). The average number of people enrolled at one point was 32 (standard deviation=50), with a median of 10.

More recently, efforts to integrate SEd programs with SE have been led out of the specialty mental health system (e.g., Killackey, Jackson, & McGorry, 2008; Nuechterlein et al., 2008a), with services often offered both on-site, in free-standing mental health agencies, or with mobile support functions. Moreover, a recent review by Smith-Osborne (2012a) described almost 15 different SEd programs providing education services and supports to veterans.

3.3.4. Supported Education Program Service Array

In practice, SEd program service features vary widely. In 2004, Waghorn & colleagues identified ten features of SEd programs. In our review of the SEd literature, we continue to find these core services offered within the context of SEd programs. In addition to these ten service components, some SEd program models now offer post-graduation employment transition support (e.g., Hutchinson et al., 2007) and work with family members to increase program engagement (e.g., Nuechterlein et al., 2008a). The ten features of SEd programs are as follows:

  1. Service coordination with professionals outside of the SEd program.

  2. Specialized career counseling, including vocational planning and exploration.

  3. Specialized, program-trained staff with time allocated explicitly to SEd programs.

  4. Financial assistance.

  5. Skill building to facilitate integration into the academic environment, including stress and time management and academic or study skills training.

  6. On-campus information about student rights and resources.

  7. On-campus or off-campus mentoring and support, individual or group support, or peer support.

  8. Coordination with post-secondary education institutions to facilitate course access or within-course assistance.

  9. Access to tutoring, library assistance, and other forms of supplemental educational support.

  10. General support (off-campus preferred) for the multiple individual barriers and life stressors that can lead to educational attrition.

A particularly common element for most SEd interventions is the presence of an individual whose job is to focus on educational goals, sometimes called an education specialist. This individual works with the program participants to identify educational goals, assist in enrolling appropriate courses, and follows up with participants to troubleshoot problems and offer supports over the course of their study (Ellison et al., 2013; SAMHSA, 2011). This education specialist-type service is most often paired with more general mentoring and support and skill building activities. Unfortunately, research on SEd interventions has not progressed to the point of being able to offer explicit guidance about what type of a SEd service array is most appropriate for clients with certain needs profiles and educational goals.

Some recently published studies have considered mechanisms and services to enhance the impact of SEd programs. One particularly interesting approach has been led by Kidd & colleagues (2012a, 2012b, 2014). This team has conducted a series of trials to examine the impact of supplementing a SEd program (RTE) with a cognitive remediation program for young adults. Cognitive remediation is a type of treatment intended to improve difficulties with attention, memory, information processing speed, problem-solving, organization, and planning. The SEd program includes remedial skills training in English fluency, study skills, and other noncredit courses. Counselors also assist students with learning difficulties and stress management. The cognitive remediation program lasts 10 weeks, with 20 computer-based, 45-minute cognitive exercise sessions held twice per week using the COGPACK program. COGPACK is a computerized remediation program targeting improved executive functioning, such as verbal learning and processing speed, among individuals with schizophrenia. COGPACK sessions cover attention, psychomotor speed, learning and memory, and executive functions. In the RCT comparing SEd alone versus SEd with cognitive remediation, there was no evidence that cognitive remediation facilitated improvement in cognition above and beyond gains in sustained attention and vigilance associated with SEd alone (Kidd et al., 2014).

Components of the Supported Education Fidelity Scale and scoresheet examine many key aspects of the potential SEd service array, including individualized post-secondary school enrollment supports, resources for students enrolled in academic institutions, knowledge-building activities, the establishment of an educational assessment and goal-setting process (see SAMHSA [2011] for more complete operationalized definitions of each component). The presence of this tool will allow future SEd evaluation and research protocols to better account for variation in SEd program service arrays.

3.3.5. Integration of Supported Education and Supported Employment Program Approaches

Researchers (e.g., Evans & Bond, 2008) have suggested that SE models may be appropriate service delivery mechanisms for providing SEd services. Attempts to integrate SEd and SE service models, particularly within mental health centers, represent a recent shift and emerging area of SEd research. Some recent publications include specific examples of integrating SEd principles and services into SE approaches. Example programs vary from basic training in Microsoft Office-type computer skills (Hutchinson et al., 2007) to a more fully integrated SEd/SE approach (Nuechterlein et al., 2008a). These models typically take place within a mental health agency, in the context of Individual Placement and Support (IPS), and with young adults with psychotic or related disorders (Rinaldi, Perkins, McNeil, Hickman, & Singh, 2010; Robson et al., 2010). The IPS model was designed as a standardized approach to SE for individuals with SMI (Drake, 1998). It consists of six evidence-based principles for SE or SEd, which are as follows: a goal of competitive employment (or educational attainment for SEd), rapid job search (or rapid enrollment in school for SEd), integration of rehabilitation and mental health, attention to consumer preferences, continuous and comprehensive assessment, and time-unlimited support (Bond, 1998). There are now two RCTs designed to examine the impact of a SEd program integrated or combined with SE (specifically IPS) compared with usual services (Killackey et al., 2008; Nuechterlein et al., 2008a). Only preliminary outcomes are available at this time; other results will be forthcoming.

Hutchinson & colleagues (2007) describe the Training for the Future program at Boston University's Center for Psychiatric Rehabilitation. This program offers a 10-month, classroom-based program that teaches computer skills. After completing the program, students participate in a 2-month unpaid internship program while taking a seminar focused on work skills. After the internship, students are provided with individual job development and employment support for as long as needed. In a repeated measures, time series pre/post evaluation design (with measurements at baseline and 3-month, 6-month, 12-month, and 18-month follow-up), this program approach demonstrated increases in participants working for pay or as volunteers from baseline to 18 months, increases in hours or work per week, and increases in mean earnings per month (among working participants). The program also found a significant linear decrease in program participants' report of mental health and rehabilitation services used over time (Hutchinson et al., 2007). Participants also reported positive gains over time in standardized measures of self-esteem and empowerment.

In a more comprehensive, integrated approach, Nuechterlein, Subotnik, Turner, & colleagues (2008a) describe an interesting model in which the IPS/SE model is being extended to include SEd for individuals with first-onset psychosis. Extending SE models to include SEd may be particularly critical for transition-age youth and young adults with first-onset mental illness. The next section of this report provides more information on extending SE models to include SEd for this very specific subpopulation of young adults.

Combined SEd and SE approaches may be more common than originally realized. Manthey, Holter, & colleagues (2012b) conducted a survey of IPS/SE programs to understand which elements of SEd services were perceived as valuable and what educational services were being provided by the programs. IPS program respondents most highly valued the provision of concrete educational services and services to minimize educational barriers for program participants. The majority of programs surveyed (approximately 57%) provided some type of educational service and support. The authors suggest that the number of SEd services provided by IPS/SE programs may have been underestimated by previous SEd-oriented surveys (e.g., Mowbray et al., 2003a) because these programs were not formally being called out explicitly as SEd programs. Integrated SEd and IPS/SE services may be feasible and may enhance the impact of either approach offered in isolation. Outcomes such as those that will be produced by the larger trial being conducted by Nuechterlein, Subotnik, Turner, & colleagues (2008a) will be helpful in understanding the impact of this combined approach.

Supported Education for First-Episode Psychosis

A combined SEd and SE model in first-episode psychosis cases may help to intervene more effectively and prevent chronic work disability status. Given that the first episode of schizophrenia typically occurs from the late teens to mid-20s, it is common that this episode will interrupt an ongoing educational experience. Nuechterlein, Subotnik, Turner, & colleagues (2008a) argue that it is a logical and developmentally appropriate step to resume an educational goal for participants who desire to do so. These desires can be seen in the Nuechterlein et al. (2008a) SEd/SE treatment group choices: 36% chose to pursue school alone, 31% chose to pursue jobs alone, and 33% chose to return to both school and jobs (most typically starting with school and adding a part-time job).

The combined IPS/SEd model tested by Nuechterlein et al. (2008a) includes a preliminary evaluation of the participant's employment or educational goals; a specialist who works to find placement either in an educational or employment setting; and support services during the participant's course of study to provide coordination with teachers, course planning, and study skills aid. These SEd services occur in tandem with more traditional SE activities. Nuechterlein, Subotnik, Turner, & colleagues (2008a) importantly note that traditional SE approaches have most often focused on chronically ill individuals. The program also encourages participants to tailor school and work to their preferences and abilities--20% choose GED credentialing programs, 60% chose community colleges, and 20% choose 4-year colleges. Preliminary findings suggest that 83% of people with recent-onset schizophrenia who received the intervention had returned to regular paid work or school during 6 months of intensive treatment as compared with 41% in the control group. Outcomes from this RCT are currently being analyzed, with results forthcoming. This approach represents a promising adaptation of SEd for first-episode mental illness.

Killackey, Jackson, & McGorry (2008) conducted a small RCT (sample of 41 individuals) that integrated some features of SEd into an SE program for people with first-episode psychosis. Program developers indicated that some integration of education components into the SE were merited due to the fact that many participants with first-episode psychosis had their educational experiences interrupted. Many program participants described having educational goals either separate from or in addition to employment goals. Killackey et al. (2008) found that this intervention approach led to greater employment and more class completion than usual care. Follow-up analyses to this study showed that no individual-level characteristics were associated with employment and education outcomes other than the program (SEd/SE vs. usual vocational and educational services) assignment (Baksheev, Allott, Jackson, McGorry, & Killackey, 2012). Education outcomes here were described as "studying or entering a course of education." Although the sample size was small, this study and those previously described provide suggestive evidence that an integrated SEd/SE approach may be helpful, particularly for those experiencing a first psychotic episode. Unfortunately, these programs do not describe the process of integrating SEd and SE interventions into one approach. Consequently, it is difficult to understand which SEd services components are specifically incorporated into SE intervention approaches and how these are implemented in the field.

3.3.6. Supported Education Program Financing

Historically, funding for SEd services has been from a mixture of federal, state, local, and foundation sources. Primary funding sources tend, in part, to be driven by the SEd program setting and owning organization. For example, in the survey by Mowbray & colleagues (2003a), most clubhouses received funding from the state or county mental health agency. Secondary funding sources from clubhouses were often VR dollars and foundation grant funding and were generated through independent fundraising. Meanwhile, on-site models in the Mowbray & colleagues (2003a) survey received funding from even more sources, including colleges or universities, state/county/city mental health agencies, VR, foundations, and United Way. All of the free-standing programs received largely mental health funding.

As detailed by Holter & Paul (2004), acquiring state education funding for SEd programs is particularly complicated. Education funding is typically divided between the U.S. Department of Education for kindergarten through 12th grade services (necessary if a SEd program provides GED service support) and the state Board of Regents (necessary if a SEd program provides adult education support). The U.S. Department of Education typically issues payments based on the headcount of students on a single day of the school year; meanwhile, the state Board of Regents may have a lengthy application that results in a calculated funding formula. Funding for special education can flow through both sources. Programs at locations such as clubhouses are not easily categorized into a secondary or post-secondary institution framework, so state funding is extremely difficult to access.

The complex funding strategies necessary to support SEd programs over time can be seen in a few published program histories. For instance, the MSERP was initially federally funded for 3 years and then moved to a combination of state and local mental health agency funding (Collins et al., 1998). In another example, Hain & Gioia (2004) describe the complicated funding history of the SEER program in Spokane, Washington. The State of Washington originally had a mandate indicating that SEER be dually maintained and funded by the Division of Vocational Rehabilitation and the local mental health community. However, original program funding was even broader based--provided by the community college system, the public mental health system, VR, and the state mental health division. Over time, however, many of these funding sources disappeared; at the time of the article's publication, 70% of program funds were from the community college system, and 30% were from the county public mental health system (Hain & Gioia, 2004). Even when SEd program publications do not describe funding sources in detail, authors often describe funding issues as an implementation and sustainability challenge.

In 2014, Manthey, Goscha, & colleagues described seven ways in which SEd programs strive to create service funding:

  1. Reallocate resources from other programs to provide services.

  2. Braid funding from municipal, federal, state, collegiate, and private corporations.

  3. Secure grant funding for short-term support while deferring costs through cross-agency collaboration.

  4. Perform general fundraising activities.

  5. Defray SEd program costs by subsidizing SEd through SE funds.

  6. Use fee-for-service schemes.

  7. Fund the program through Community Mental Health Services Block Grants.

To facilitate funding for SEd services, Manthey, Goscha, & colleagues (2014) recommend that funders lift some key funding barriers to help ease SEd program implementation and dissemination: (1) remove caps on billable hours for SEd services; (2) create guidelines to allow specialty mental health centers to bill Medicaid for SEd; (3) create specific guidelines to allow SEd programs to be billed as part of SE services; and (4) encourage increased use of peer support-run SEd services while allowing SEd services to be billed through peer support channels. Sustainable and consistent funding sources continue to impede program growth and evaluation.

3.3.7. Other Programs Similar to Supported Education in the Literature

Most standalone post-secondary education interventions for individuals with psychiatric disabilities reference their approach specifically as "supported education." However, we did find some publications with interventions targeting secondary students (high school, transition-age youth) where the approach was not necessarily defined as "supported education," but where the intervention had a similar service array. For instance, the Portland Identification and Early Referral (PIER) program focuses on helping secondary school students with psychiatric disabilities and their family members better understand mental illness. The program describes the use of strategies to help students complete secondary education and enter post-secondary education settings or employment (Downing, 2006). Another secondary school approach administered occupational therapy in the public school system for children with emotional disturbances. That approach was designed to enhance learning and promote high school degree completion (Chandler, 2007). We did not extensively review these types of secondary school approaches; however, we wanted to note their presence in the literature.

Another way in which SEd approaches are noted in the literature, but not explicitly labeled as "SEd programs," was when these approaches were included in a very broad array of integrated education and employment support services. Many early intervention programs for individuals experiencing a first episode of psychosis include SEd components, without explicitly being named as SEd programs. For example, a multisite RCT is currently being conducted by McFarlane & colleagues (McFarlane et al., 2014) involving young adults at risk for schizophrenia and psychosis. This trial is designed to examine the impact of the Early Detection and Intervention for the Prevention of Psychosis Program (EDIPPP) (McFarlane et al., 2012), which examines the effectiveness of a PIER-based program across the United States. EDIPPP includes a SEd program that is bundled with an array of other family-based services and supports (McFarlane et al., 2014). The focus of this intervention is on the early identification, treatment, and prevention of psychosis among young adults (and not solely post-secondary education enrollment). Consequently, this type of trial does not explicitly examine SEd program outcomes, but represents the integration of SEd approaches into a broader mental health intervention.

3.3.8. Implementation Challenges

Issues related to program implementation are often described in published SEd research and evaluation studies. Client-level implementation issues and challenges include participation, attrition, and hardships facilitating professor-student relationships. Barriers to program participation were more commonly described by program participants with moderate participation rates than individuals with high program participation rates (as cited in Rogers, Kash-MacDonald et al., 2010). Participation rates vary related to participants' substance abuse behaviors, number of hours worked for pay, quality of life, and size of social network (as cited in Rogers, Kash-MacDonald et al., 2010). Positive client-level outcomes result when program staff are able to facilitate effective partnerships between students and their instructors (Cook & Solomon, 1993). The ability of the case worker to disclose as generally as possible about the student to the professor enhanced the chances of school success (Nuechterlein et al., 2008a). The number of staff providing mobile support per client may also need to be considered; as Cook & Solomon (1993) noted, more than one staff person is needed to provide adequate mobile support.

SEd program systems-level implementation issues and challenges are also described in research and program evaluation publications. These issues largely focus on developing a positive working relationship with the staff or faculty within the school community where services are provided and building up the capacity of mental health services at the educational institution, enabling students to have strong contact with their mental health services provider. Suggestions for building relationships with educational institutions include conducting in-service training and liaison activities with post-secondary faculty (Cook & Solomon, 1993); having a positive relationship with the representative of the community college, which is necessary to maintain the SEd program as a high priority (Mowbray, 2000); contacting the disabled student services office before the start of the first day of class, instead of waiting for a problem to arise (Nuechterlein et al., 2008a); and collaborating with consumer-run programs and regularly presenting about the SEd program and what it can do for its clients (Mowbray, 2000). Most educational institutions are ill equipped to provide the treatment and resources that students with SMI require, so mental health programs need to initiate SEd programs (as cited in Unger, 2011; Unger, Pfaltzgraph et al., 2010; Wagner & Newman, 2012) because students who are able to maintain contact with mental health services have a higher retention rate than those who are unable to maintain contact (as cited in Unger, 2011; Watkins, Hunt, & Eisenberg, 2012).

Ellison & colleagues (Ellison et al., 2014) describe program modifications that were made to add a SEd component to an IPS-SE model for implementation with an emerging adult population (17-20 years). In particular, early feasibility testing revealed the need to have a separate educational specialist position (in addition to the already existing employment specialist). The program offered both an education and employment-oriented program track; however, enrollment in the education track was below expectations. An education specialist was added to be a resource for education-related needs; SEd program participation increased. This program also used peer mentors but noted challenges in keeping peer mentors consistently employed. They eventually went with older peer mentors (ages 28 and 30) "who had lived experience, but were far enough along in their own development and recovery to maintain strong boundaries with participants" (Ellison et al., 2014).

3.4. Synthesis of Prior Review Findings on the Impact of Supported Education Interventions

Three particularly comprehensive reviews of SEd studies have been published. The first review was written by Mowbray & Collins (2002) and summarized publications up to 1996. The second review was published by Leonard & Bruer (2007), with a particular focus on implications for psychiatric hospitals and other mental health facilities. Most recently, Rogers, Kash-MacDonald, Bruker, & Maru (2010) conducted a systematic review of SEd publications from 1989 to 2009. Rogers &d colleagues (2010) focused on study designs intended to examine the impact of SEd programs. Interestingly, the Leonard & Bruer (2007) publication included no papers authored by Mowbray & colleagues. Meanwhile, Dr. Mowbray authored or co-authored seven of the 13 articles reviewed by Rogers & colleagues (2010). The Rogers review is the most recent and by far the most systematic; consequently, this review is summarized as follows.

3.4.1. Articles Reviewed by Rogers and Colleagues (2010)

Rogers & colleagues (2010) summarized the results of 13 articles published between 1989 and 2009. All articles were reviewed by three individuals and separately rated for the quality of their research methods. The review article individually summarizes each article: its findings and its methodological strengths and weaknesses. Seven of the 12 publications reviewed by Rogers & colleagues (2010) were conducted by Mowbray & colleagues using the MSERP study dataset. In fact, Rogers & colleagues (2010) note that the number of articles published on the MSERP dataset skews the findings toward one model and obscures "the number of alternative models which have not been adequately tested" (p. 8). The articles included in Rogers & colleagues' (2010) review and their associated study designs are described in Table 3-2.

TABLE 3-2. Study Designs for Publications Reviewed by Rogers & Colleagues (2010)
Study Design Program Name Citations
Experimental (RCT) MSERP Collins et al., 1998; Collins, Mowbray, & Bybee, 1999a, 1999b
Quasi-experimental (comparison group) Program not named Hoffmann & Mastrianni, 1993
Correlational MSERP Collins et al., 1999b; Collins, Mowbray, & Bybee, 2000; Mowbray, Bybee, & Collins, 2001; Mowbray et al., 1996
Pre/Post Continuing Education Project, Thresholds Community Scholars Program, others not named Best, Still, & Cameron, 2008; Cook & Solomon, 1993; Unger, Anthony, Sciarappa, & Rogers, 1991; Unger & Pardee, 2002; Unger et al., 2000

3.4.2. Seminal Experimental or Quasi-Experimental Studies Published before 2010

The systematic review conducted by Rogers & colleagues (2010) found only two SEd trials that the researchers considered "rigorous": one an experimental RCT (Collins et al., 1998) and the other a high-quality quasi-experimental trial (Hoffmann & Mastrianni, 1993). These studies continue to stand as seminal works in the field.

Collins, Bybee, & Mowbray (1998). The only RCT of SEd was published by Collins, Bybee, & Mowbray (1998). This study included 397 participants. Participants were recruited from the Detroit metropolitan area and primarily came from the public mental health system. Some came from self-help programs and advocacy services, and others were recruited by word of mouth. These participants were enrolled in one of two experimental conditions (a classroom intervention and a group support intervention) or a control condition (where individuals were given the name of a support person to contact with questions). Both the classroom and group model had meetings twice a week for 2.5-hour sessions (for 14 weeks). The classroom model had two instructors and a curriculum that covered managing the campus environment, career exploration, and managing stress. The group model had two facilitators; one was a mental health consumer. Groups were designed to explore career and education options and make meaningful, individualized decisions. All participants received an information packet covering assistance in obtaining VR services, facilitated access to special student services and advising, on-site mentorship, and access to contingency funds for assistance with short-term, school-related expenses.

Collins & colleagues (1998) found that participant satisfaction was significantly higher among those participating in the group model than among those in the control group. Participation did vary significantly across the three groups, with the highest participation rates in the group model condition. Authors examined participation rates and found that 35% of those with high participation rates in SEd programs enrolled in college or vocational services compared with 23% of those with no participation in SEd, a significant positive effect (Collins et al., 1998). This was the only significant finding that resulted from comparisons across the three groups. There were nonsignificant differences among the three conditions on having taken college or vocational education classes since baseline and on work status. In a long-term follow-up of this same sample, the percentage employed or enrolled in school increased significantly, from 24% to 39%, for those in a classroom SEd model (Mowbray et al., 1999).

Unfortunately, Mowbray & colleagues' (1999) study failed to use an intent-to-treat model and instead analyzed only the data available from those subjects who completed the post-test assessment. This is particularly problematic because the study had 26% of participants (104 individuals) drop out between baseline and post-test. The participants who dropped out during the course of the trial are also not separated by condition (two treatment conditions versus control). This methodological flaw makes the study's outcomes difficult to interpret.

Hoffman & Mastrianni (1993). The only quasi-experimental study of a SEd program published before 2010 examined a SEd intervention conducted within an inpatient psychiatric hospital. Hoffman & Mastrianni (1993) compared the outcomes of participants in this SEd program with those of patients from a matched psychiatric hospital with a more traditional approach to inpatient treatment. The SEd program integrated academic goals and opportunities into those typically available in regular treatment. Individuals in the SEd program also participated in special academic activities in partnership with a community college. The SEd intervention group had a higher rate of college enrollment (69%) than did the comparison group (47%). And, of those who enrolled in college, SEd participants (88%) were more likely than the comparison group (58%) to return to school full-time or progress from part-time to full-time in school. Unfortunately, this study had several methodological problems. First, subjects in this study were not randomized to treatment conditions, and there were some notable differences between groups. For example, 37% of the participants in the comparison group had primary Axis II diagnoses (i.e., personality disorders) compared with 0% in the experimental group. Analyses also focused on only post-test data, without controlling for baseline levels of the key outcome variables. Finally, subjects enrolled in both the intervention and comparison groups had particularly high levels of baseline education (average of 13 years for both groups), so it is unclear how these results would translate to a more typical inpatient psychiatric population.

3.4.3. Rogers & Colleagues' (2010) Conclusions

Reflecting on the state of the literature regarding the impact of SEd, study results suggested that SEd programs may help increase college enrollment and vocational outcomes (e.g., Mowbray et al., 1999; Unger et al., 1991), improve school retention rates (e.g., Unger et al., 2000), and possibly decrease psychiatric hospitalizations (Unger et al., 1991). More specifically, Rogers & colleagues (2010) drew the following positive conclusions about the impact of SEd programs:

  • There is suggestive evidence (from noncontrolled studies) that individuals improve their employment and educational status after participating in a SEd program (Best et al., 2008; Cook & Solomon, 1993; Hoffmann & Mastrianni, 1993; Unger et al., 1991; Unger & Pardee, 2002; Unger et al., 2000).

  • Individuals who stay engaged in SEd interventions appear to be able to finish courses and keep satisfactory grades (Best et al., 2008; Cook & Solomon, 1993; Unger & Pardee, 2002; Unger et al., 2000).

Unfortunately, Rogers & colleagues (2010) also came to the following conclusions:

  • There is no comparative evidence that participation in a SEd program leads to gains in post-secondary educational enrollment or employment when compared with the outcomes of individuals not participating in a SEd program (Mowbray et al., 1999).

  • No significant quantifiable changes in self-esteem or quality of life resulted after participation in a SEd program (Unger & Pardee, 2002; Unger et al., 2000).

  • Effectiveness data in support of SEd programs are limited. This is due to the absence of well-controlled studies, the limited number of studies that examined key outcomes of interest (e.g., degree completion, employment), and the preponderance of short follow-up periods limiting the ability to examine longer-term participant outcomes.

3.5. Impact of Supported Education Interventions

When Rogers & colleagues (2010) conducted their systematic review of SEd programs, they identified 17 published outcome studies that included pre/post (n=4), experimental (n=3), quasi-experimental (n=1), correlational/survey/observational (n=9), or post-test only (n=4) designs. Seven of these 13 manuscripts were published by the same researcher (Mowbray). Our review of articles published prior to the fall of 2015 uncovered an additional 16 outcome studies published since Rogers & colleagues' (2010) review. These 16 studies included the following designs: pre/post (n=6), experimental (n=5), quasi-experimental (n=0), correlational/survey/observational (n=4), and post-test only (n=1) designs. The four experimental study publications represent three different RCTs, two of which are ongoing and do not yet have extensive published results reflecting comparative outcomes. The number of SEd program outcome studies accumulated from 1989 to 2009 almost doubled in the last 5 years (2010-2014). This represents marked growth in the literature. Perhaps more importantly, these publications also demonstrate the emergence of new scientists in the field of SEd research. These 16 recent publications also represent the work of 13 different first authors. A list of these 16 studies, their research designs, and types of outcomes reported in each publication can be found in Table 3-3. In addition to these 16 studies, we also found 15 publications that were not outcome-oriented trials: ten review articles (or calls for future research or opinion papers), three descriptive program summaries, and two other miscellaneous papers (a SEd guide and an environmental scan). Findings reported in these studies will also be included in this portion of the literature review report.

In this section, we will summarize both client-level and systems-level outcomes described in these original research and review publications. We also include comments about previous findings published before 2010. We did not systematically rate these studies by the merits of their analytic designs; however, in Section 3.6, we discuss methodological strengths and weaknesses of the research studies examined.

TABLE 3-3. Outcomes Examined in SEd Program Impact Studies Published since Rogers & Colleagues' (2010) Systematic Review
Articles Article
Type
Study
Design
Educational
Engagement
Educational
Attainment
Employment Health and
Mental
Health
Self-
Perception
Service
Access
Baksheev, Allott, Jackson, McGorry, & Killackey (2012) Original research RCT X   X      
Kidd, Kaur-Bajwa et al. (2012b) Original research Pre/post without comparison group X   X      
Ellison, Klodnick, Bond, Krzos, Kaiser, Fagan, & Davis (2014) Original research Pre/post without comparison group X X        
Kidd, Kaur et al. (2014) Original research RCT       X X  
Kidd, Kaur-Bajwa et al. (2012a) Original research Pre/post without comparison group       X    
Manthey, Holter et al. (2012b) Original research Survey       X   X
Nuechterlein, Subotnik, Turner et al. (2008a) Program summary RCT in progress (only outcomes measured described, not comparative data) X   X      
Nuechterlein, Subotnik, Ventura et al. (2008a) Original research RCT (conference presentation summary) X   X      
Rinaldi, Perkins et al. (2010) Original research Pre/post without comparison group X   X      
Robson, Waghorn et al. (2010) Original research Pre/post without comparison group X X        
Schindler, & Sauerwald (2013) Original research Pre/post without comparison group X   X X X  
Smith-Osborne (2012a, 2012b) Program summaries RCT in progress (no comparative outcomes published to date)   X   X X X
Thompson (2013) Original research Post-test only         X  
Wagner & Newman (2012) Original research Survey X X X      
Watkins, Hunt, & Eisenberg (2012) Original research Qualitative            
Yahaya, Ramli et al. (2010) Original research Correlational            

3.5.1. Client-Level Outcomes

Rogers & colleagues (2010) noted that typical processes and outcomes described in studies of SEd programs included:

  • Educational Enrollment or Engagement: Enrollment in post-secondary education programs, classes, or courses

  • Educational Attainment: Post-secondary courses completed, grades attained, certificates achieved, or diplomas or degrees.

  • Employment: Full-time or part-time (paid or volunteer) work, hours worked, and wages earned.

  • Self-Perception: Self-esteem, self-efficacy, quality of life, and adjustment.

  • Number of psychiatric hospitalizations.

  • Consumer satisfaction.

After examining the frequency of these outcomes across the 14 recently published studies, we added or modified a few outcomes of interest for our review:

  • Health and Mental Health Status: Cognitive and executive functioning, general physical health, and specific psychiatric symptoms (e.g., post-traumatic stress disorder [PTSD], psychotic symptoms).

  • Noneducation and Employment-Related Service Use: Use of and engagement in mental health services, including psychiatric hospitalizations.

Similar to the findings from Rogers & colleagues (2010), the most commonly reported outcome within studies published since 2010 was educational engagement and then employment. Only a few articles reported on any type of educational attainment outcome. None reported degree status achieved (beyond the receipt of a program certificate). Several articles reported on health/mental health status, as well as self-perception outcomes. Findings related to these outcomes from studies published both before and after 2010 are described as follows.

Educational Enrollment or Engagement

Secondary data analysis from the National Longitudinal Transition Studies (Wagner & Newman, 2012) has found that school completion rates for students with emotional disturbances who are enrolled in special education services have increased from 47% to 78% from 1990 to 2005. Furthermore, the percentage of students with SEd who enroll in post-secondary education has increased nationwide from 18% to 35%. This increase in high school educational attainment and post-secondary educational enrollment shows the need for SEd programming to meet the needs of these students as they increasingly progress into post-secondary education institutions (Kirsh et al., 2014). As discussed frequently in the literature, both educational enrollment and attainment continue to be critical outcomes to monitor within SEd programs.

Approximately half of the articles published since 2010 mentioned any type of educational enrollment or engagement outcome. The majority of articles mentioned educational enrollment outcomes related to SEd program participation or post-secondary course enrollment.

Earlier work using an RCT found nonsignificant differences between the two treatment and one control conditions on having taken college or vocational education classes since baseline (Collins et al., 1998). In this same study, greater SEd program participation was related to greater participation in college or vocational classes (Collins et al., 1998). More recently, there is suggestive evidence (where significance of the outcomes was not indicated) that SEd program participants have increased enrollment in post-secondary educational institutions and courses (Kidd et al., 2012b; Manthey et al., 2014; Mowbray, 2000). Furthermore, protective factors that help to retain students with SMI enrolled in post-secondary education have been described in the literature. These include active coping, peer support, counseling and psychosocial support, academic support, and academic accommodations (as cited in Hartley, 2010).

Many studies combined educational outcomes with employment outcomes, with many reporting significant increases in educational engagement or employment (Rinaldi et al., 2010; Schindler & Sauerwald, 2013; SAMHSA, 2011; Unger et al., 1991--as cited in Unger, 2011). Killackey et al. (2008) found that an integrated SEd/SE intervention approach led to greater employment and more class completion than usual care. With combined education/employment outcomes, it is impossible to discern the differential impact of the intervention on only education versus employment outcomes. Sometimes this choice by study authors may be due to an integrated SEd/SE approach where program goals were education or employment (but not both goals for all participants). However, some caution should be taken here in interpreting studies that combine educational enrollment and employment outcomes. Many studies have very small sample sizes and may be underpowered to detect group differences. Outcomes may have been combined due to neither outcome alone resulting in significant differences. For example, this was true for the study conducted by Schindler & Sauerwald (2013), where nonsignificant changes occurred in enrollment in higher education from pre-test to post-test.

In one of the few recent RCTs involving a SEd intervention component, Nuechterlein, Subotnik, Ventura, and colleagues (2008a) found that 83% of subjects in the intervention group (combined SEd and SE) returned to work or school, compared with 41% of those in the treatment-as-usual group (p<0.001) during the first 6 months of treatment. This pattern continued even at the end of the 18-month trial (72% versus 42%). Nuechterlein, Subotnik, Ventura, and colleagues' (2008a) study likely combined employment and education outcomes because both were targets of the combined intervention approach. Unfortunately, this combination does not allow the separate examination of the additive impact of the SEd program component within an SE approach. Studies are needed to examine SE alone plus SE with SEd to understand the unique and differential impact of each approach on client outcomes.

Educational Attainment

One-third of articles reviewed mentioned educational attainment as an outcome. Three of the 16 outcome-oriented studies published since 2010 reported on educational attainment. The variables used to asses educational attainment included course completion (as cited in Rogers, Kash-MacDonald et al., 2010; as cited in Unger, 2011; Cook & Solomon, 1993; as cited in Manthey, Goscha et al. 2014; as cited in Mueser & Cook, 2012; Robson, Waghorn et al., 2010), post-secondary degree/certificate completion (as cited in Unger, 2011; Morrison, Clift et al., 2010), high school degree completion (Wagner & Newman, 2012; as cited in Ellison, Rogers, & Costa, 2013; Ellison, Vorheis et al., 2014), satisfactory GPA (Smith-Osborne, 2012b; as cited in Rogers, Kash-MacDonald et al. 2010; as cited in Unger, 2011), and number of credit hours enrolled in and completed (Unger, Pfaltzgraph et al., 2010; as cited in Unger, 2011). Severity of illness is often mentioned as the first barrier to degree/certificate program completion; however, environmental supports have been proven to prevent educational attrition due to mental illness (as cited in Unger, 2011).

Studies generally reported increases in educational attainment from pre-intervention to post-intervention; however, no studies reported these gains in comparison to a group not involved in a SEd program. For example, Robson, Waghorn, & colleagues (2010), in a pre/post comparison of an IPS plus SEd program, reported that 70% of their program's participants had completed their course of study or were continuing with their studies at an 18-month follow-up. In another example, Smith-Osborne (2012a) reports increased GPAs post-SEd intervention among the veterans participating in an RCT; however, GPAs for the intervention group when compared with the control group have not yet been published.

Degree completion was the most rarely reported indicator of educational attainment. The small number of original research articles measuring educational attainment through degree or certificate completion is likely affected by relatively short follow-up data collection periods that do not extend far enough to capture degree/certificate completion. Limited research funding may prohibit the longer-term follow-up periods necessary to examine degree completion. The use of variables measuring course completion, credit completion, or GPA are useful for measuring short-term changes from pre-test to post-test that can indicate potential successful degree/certificate completion. However, studies that follow participants for several years post-intervention to understand the impact on degree completion are very much needed.

Employment

Approximately half of the articles reviewed mentioned some type of employment outcome. The majority of these (11) are from articles published since 2010 or later. The variables used to assess employment outcomes included currently employed (e.g., Collins, Bybee, & Mowbray, 1998; as cited in Krupa & Chen, 2013; as cited in Manthey, Goscha et al., 2014; as cited in Morrison, Clift et al., 2010; Mowbray, 2000; Rinaldi, Perkins et al., 2010; as cited in Rogers, Kash-MacDonald et al., 2010; Schindler & Sauerwald, 2013; Unger, Pfaltzgraph et al., 2010; Wagner & Newman; 2012), type of employment (Cook & Solomon, 1993; as cited in Unger, 2011), hours worked (Cook & Solomon, 1993), pay (Cook & Solomon, 1993; as cited in Manthey, Goscha et al., 2014), and job tenure (Kidd, Kaur-Bajwa et al., 2012a; as cited in Krupa & Chen, 2013). Some of the articles reported the combined educational and employment outcomes together (Baksheev, Allott, Jackson, McGorry, & Killackey, 2012; Mowbray, 2000; as cited in Rogers, Kash-MacDonald et al., 2010; Schindler & Sauerwald, 2013).

The majority of these articles reported positive, if not significant, employment gains in all of the variables used to measure this outcome. Two relatively recent studies are worth mentioning, specifically because of their relatively strong research designs. In a repeated measures, time series pre/post evaluation design (with measurements at baseline and 3-month, 6-month, 12-month and 18-month follow-up), Hutchinson & colleagues (2007) tested the impact of a SEd program (focused on computer skills training) and an SE program. This program approach demonstrated increases in participants working for pay or as volunteers from baseline to 18 months (18%-64%), increases in hours or work per week, and increases in mean earnings per month (among working participants) (Hutchinson et al., 2007). Notably, Hutchinson & colleagues (2007) carefully accounted for attrition over time in all analyses. Gutman, Kerner, & colleagues (2009) conducted a small sample RCT (n=38) that included a treatment-as-usual control group. The SEd program included a 12-week on-site classroom training program that also included academic mentoring and support. Gutman & colleagues (2009) found that 63% of participants at the 6-month follow-up were enrolled in some form of educational program or job training, had obtained employment, or were in the process of applying to a specific program in the next year. This outcome was true for only 6% of participants in the control group.

Some more recent studies have explored the impact of educational attainment on employment (as cited in Ennals, Fossey et al., 2014). This exploration is an important step in determining whether SEd outcomes not only lead to educational attainment but subsequently lead to better employment outcomes as well. There may be active disincentives for SEd program participants to seek employment. For example, Krupa & Chen (2013) reviewed research stating that a disincentive to employment for individuals in SEd programs can be the risk of losing government financial assistance. It is important to understand the perceived barriers to reaching educational or employment goals for program participants.

National estimates of employment for individuals with mental illness is at 48%, SMI at 37%, and schizophrenia and related disorders at 22% (as cited in Unger, 2011). These employment rates are significantly lower than that of the general population (Wagner & Newman, 2012).

Self-Perception

Approximately one-third of the articles reviewed mentioned some type of self-perception outcome. Variables with reported self-perception outcomes include school efficacy, self-esteem, coping, anxiety, empowerment, recovery/resilience, psychosocial wellness, quality of life, social adjustment, social support, and self-efficacy. All but a few articles mentioned increased or significant outcome measures on all variables measuring self-perception (as summarized by Manthey, Goscha et al., 2014; Thompson, 2013; Smith-Osborne, 2012). Of note, Hutchinson & colleagues (2007) used standardized measures of self-esteem (the Tennessee Self-Concept Scale) and empowerment (Empowerment Scale). Their repeated measures design demonstrated significant linear increases in ratings on these two scales over the 18-month course of the study. These results are confirmed via qualitative findings. In qualitative studies, SEd program participants consistently report an increased sense of control, empowerment, and socialization that they gain from program participation (e.g., Bellamy & Mowbray, 1998; Schindler & Sauerwald, 2013). Nonsignificant changes in self-esteem or quality of life were found in less recent studies without standardized measures (Cook et al., 2005a; Unger & Pardee, 2002; Unger et al., 2000). The use of standardized scales in more recently published studies examining changes in self-perception as a result of SEd interventions may help to explain why recent studies are more likely to demonstrate positive change.

Health and Mental Health

Less than one-quarter of the articles reviewed mentioned any type of health or mental health outcome. All of these were from articles published since 2010. Specific health and mental health outcomes reported to be associated with SEd program participation included increases in independent living (as cited in Manthey, Goscha et al., 2014) and decreased PTSD symptoms and increased health (as cited in Smith-Osborne, 2012). Gutman & colleagues (2009) found statistically significant differences between the experimental and control group on three different rating scales measuring social skills, school behavior, and attention skills. The recent RCT by Kidd & colleagues (2012b, 2014) has supplemented a classroom-based SEd program with cognitive remediation. This trial includes many standardized measures of executive functioning. The SEd program resulted in significant improvements from pre-intervention to post-intervention in the Trail Making Test B, verbal learning as indicated in the California Verbal Learning Test (CVLT), the time component of the Digit Vigilance Test, and on the general psychosis symptomatology measure (Positive and Negative Syndrome Scale [PANSS]). Of further note, significant improvement in sustained attention and vigilance was found in only the control group that received the standard SEd program (without the cognitive remediation component) (Kidd et al., 2012b; Kidd et al., 2014). The integration of standardized measures of health, mental health, and particularly executive functioning is a positive trend in studying SEd program outcomes.

Use of Other Types of Services

Only a few articles reviewed mentioned any type of outcome related to mental health service use, access, or engagement. Two articles are worth mentioning. First, Collins, Bybee, & Mowbray (1998) found a significant difference in involvement in rehabilitative services for participants with the highest level of participation in the group condition alone. Participants with lower participation rates or participants enrolled in the classroom condition or control group were significantly less likely to be involved in rehabilitative services than those who were high participants and in the group condition. Second, Hutchinson & colleagues (2007) found a significant linear decrease in program participants' report of mental health and rehabilitation services used over the course of 18 months. Ideally, SEd program participation enhances client functioning thereby reducing the need for intensive or restrictive psychiatric treatment. Findings particularly by Hutchinson et al. (2007) are promising, but understanding the impact of SEd programs on mental health and noneducation services merits further research attention.

Impact of Client Characteristics on Program Participation and Outcomes

SEd program effects may or may not differ based on various client characteristics. Importantly, the presence or absence of major psychiatric diagnoses does not appear to affect a SEd program participant's post-secondary education enrollment (Unger & Pardee, 2002; Unger et al., 2000). A client's prior work or school activity appears to be the strongest predictor of later involvement in work and school (Collins et al., 2000). Single, unmarried SEd program participants were less likely than married participants to be involved in post-intervention work and school activities (Collins et al., 2000). More frequent contact with a social network has been found to be associated with more post-intervention work and school activities (Collins et al., 2000). Meanwhile, less financial stability was associated with fewer post-intervention work and school activities (Collins et al., 2000). Gutman & colleagues (2009) noted several other factors associated with program success: adherence to a medication routine, stable residence, and motivation to attend the program regularly. Meanwhile, diagnosis, prior educational level, number of past 5-year hospitalizations, age of mental illness onset, and parental education had no relationship to program success (Gutman et al., 2009).

Individual client characteristics also appear to be associated with SEd program participation. More hours worked per day, higher ratings of residential quality of life, and a larger social network were all related to higher attendance in a MSERP SEd group (Bybee, Bellamy, & Mowbray, 2000). The presence of a substance abuse problem, on the other hand, was associated with lower attendance in a SEd program (Bybee et al., 2000).

A few other adaptations to the traditional SEd program design have been implemented to target students with cognitive difficulties who do less well in SEd programs alone (Kidd et al., 2012b), veterans with PTSD (Smith-Osborne, 2012a), and Orthodox Jews with SMI who have educational goals unique to their religious community (Shor & Avihod, 2011). Some of these adaptations have shown promising results (Kidd et al., 2012b), some outcomes have yet to be published (Smith-Osborne, 2012a), and others may not be published beyond program summaries because the population of interest is unique, and outcome goals are not easily generalizable (Shor & Avihod, 2011).

Summary of Supported Education Program Impact on Client-Level Outcomes

In reflecting on the impact of SEd programs on client-level outcomes, we note some changes in the field in the last 5-10 years. Recent publications have added to prior evidence from noncontrolled studies demonstrating that individuals appear to improve their educational enrollment after participating in a SEd program. Recent research also adds to the suggestive evidence (from noncontrolled studies) that individuals improve their employment and educational attainment after participating in a SEd program. Unfortunately, there continues to be a lack of comparative evidence that participation in a SEd program leads to gains in post-secondary educational enrollment and more importantly, educational attainment. Randomized trials currently in progress (Nuechterlein et al., 2008a; Smith-Osborne, 2012a, 2012b) may add to this comparative literature in forthcoming publications.

The comparative evidence on the impact of SEd on both education and employment outcomes is growing, with particularly noteworthy work by Hutchinson & colleagues (2007), Nuechterlein, Subotnik, Turner, & colleagues (2008a), Killackey & colleagues (2008), Baksheev & colleagues (2012), and Gutman & colleagues (2009). The rationale to combine education and employment-oriented program approaches is compelling, particularly as many young adults jointly pursue both education and employment goals. These experimental or quasi-experimental studies do demonstrate quantifiable impacts on employment outcomes. Unfortunately, for the purpose of understanding the specific impact of SEd interventions, these studies are not as helpful. Three of the four studies test a combined SEd/SE approach where the differential impact of SEd on employment beyond the impact of the SE approach cannot be determined. But, generally, there appears to be growing research that SEd approaches, particularly when combined with SE, increase participant employment.

At the time of Rogers & colleagues' (2010) systematic review, the authors concluded that SEd studies demonstrated no significant quantifiable changes in self-esteem or quality of life after participation in a SEd program (Unger & Pardee, 2002; Unger et al., 2000). Recent research provides evidence to contradict this conclusion (outcomes summarized by Manthey, Goscha et al., 2014; Thompson, 2013; Smith-Osborne, 2012; Hutchinson et al., 2007). Qualitative studies also support these positive changes in self-perception as a result of SEd program participation (e.g., Bellamy & Mowbray, 1998; Schindler & Sauerwald, 2013).

Finally, it is premature to conclude that SEd programs affect general health, mental health, or functional status. This is hindered by the general lack of longitudinal data examining these outcomes of interest. However, the use of standardized measures to assess self-perceptions, and health and mental health status or functioning is a noteworthy advance in this field. Ongoing trials may provide new evidence about health and mental health outcomes in the next 5-10 years.

3.6. Supported Education Research Methods Summary and Gaps in the Published Literature

This review has highlighted the substantial number and variety of efforts in SEd research. The review also highlights what is missing in the field and next steps needed for SEd research. This section first summarizes study methods and then discusses gaps in the literature and potential next steps.

3.6.1. Study Methods

Outcome Data and Measures

In studying the impact of SEd interventions on client outcomes, researchers primarily rely on primary, client-reported data. Data from program or education administrators or the use of secondary, administrative data are rarely described. Increasingly, however, standardized measures of health, self-perceptions, mental health, and executive functioning are being used in SEd program outcome studies.

Variables measured using standardized outcome measures include anxiety, using the Personality Assessment Inventory (Collins et al., 1998) and the Zung Self-Rating Anxiety Scale (Cook & Solomon, 1993); social adjustment, using the Social Adjustment Scale-Self-Report (Collins et al., 1998); symptomology, using the Symptom Checklist-10 (Collins et al., 1998) and the Brief Psychiatric Rating Scale (Robson et al., 2010); drug and alcohol use, using two scales from the Personality Assessment Inventory (Collins et al., 1998); self-perception/self-esteem, using Rosenberg's Self-Esteem Scale (Collins et al., 1998; Cook & Solomon, 1993; Kidd et al., 2012a; Kidd et al., 2014); empowerment using the scale developed by Rogers, Chamberlin, Ellison, & Crean (1997); quality of life using Lehman's Quality of Life Interview (Collins et al., 1998); coping, using the coping mastery scale (Cook & Solomon, 1993); mental health, using the PANSS (Kidd et al., 2012a; Kidd et al., 2014); cognitive measures, using the Wide Range Achievement Test 3, Trail Making Test A, the digit span subtest of the Wechsler Adult Intelligence Scale-III, CVLT, Trail Making Test B, Wisconsin Card Sorting Test, and the Digit Vigilance Test (Kidd et al., 2012a; Kidd et al., 2014); and SEd program fidelity, using the SAMHSA Supported Education Fidelity Scale (Manthey et al., 2012b). Increasing use of standardized measures to understand program impact provides an improved opportunity to demonstrate quantifiable changes in client outcomes as a result of SEd program participation.

Length of Follow-Up Periods

Of the few studies that collected post-intervention outcome data, most included only a post-program completion assessment (post-test). Times between baseline/pre-test and follow-up/post-test ranged from 3 months to 9 months. This range in data collection periods obviously creates difficulties understanding immediate post-program impact. Longer programs would have allowed program participants a longer time to enroll in education courses or seek and obtain employment. Consequently, post-program participation outcomes should not be compared directly across studies.

Preliminary studies exploring ways to adapt SEd programs often only reported findings at the conclusion of the program; for example, when studying the integration of cognitive remediation into a pre-existing SEd program (Kidd et al., 2012a; Kidd et al., 2012b). Publications focused on adaptations to traditional SEd programs are new to the research literature, and outcome data with longer follow-up data collection periods on these adaptations will require more time for these outcomes to be measured, analyzed, and published.

Only a handful of articles reviewed included follow-up data collection beyond an immediate post-program assessment. Excluding post-test only data collection, seven of the original research outcomes study publications that we reviewed collected follow-up data anywhere from 8 months to 3 years after baseline. A typical post-secondary degree/certificate program takes 2-4 years to complete. With the exception of one study with a follow-up data collection period of up to 3 years, none of the follow-up data collection periods would have been positioned to gather information on post-secondary degree completion or program certification attainment for any program participants still in the midst of their course of study. Data suggests that SEd programs help participants with SMI reach their educational goals (e.g., Collins et al., 1998; Rogers et al., 2010), yet current research lacks the minimum necessary follow-up periods to accurately assess degree/certificate completion for the majority of clients. Mueser & Cook (2012) and Manthey, Goscha, & Rapp (2014) noted that short-term funding periods may limit the ability of researchers to accurately assess the primary goals of SEd programs and instead lead researchers to focus on the short-term goals of educational enrollment, class or credit completion, and GPA (e.g., as cited in Rogers, Kash-MacDonald et al., 2010; Robson, Waghorn et al., 2010; as cited in Unger, 2011; Unger, Pfaltzgraph et al., 2010).

Strengths and Weaknesses of Study Designs

Ellison, Rogers, & Costa (2013) mentioned in their review of SEd literature for young adults that many of the articles reviewed are what they considered "pre-scientific" and consequently the impact of many SEd programs have yet to be measured systematically. Many research studies include small sample sizes, limited use of control groups, short follow-up periods, use of nonstandardized measures, and preliminary research analysis. These critiques could also be named as true for some more recent publications not included in the Ellison, Rogers, & Costa (2013) or Rogers, Kash-MacDonald, & colleagues (2010) reviews (e.g., Kidd et al., 2014; Manthey et al., 2014; Manthey et al., 2012b; Robson et al., 2010; Schindler & Sauerwald, 2013). Small sample sizes (e.g., Gutman et al., 2009; Kidd et al., 2014; Robson et al., 2010) and high attrition rates (e.g., Cook & Solomon, 1993; Manthey et al., 2014) limit analysis possibilities and the generalizability of findings. Small sample sizes could explain why outcome data from original research publications and review articles do not consistently mention the statistical significance of findings (e.g., Krupa & Chen, 2013; Robson et al., 2010; Thompson, 2013). A high number of review articles or program summaries continue to highlight the importance of SEd programs, but additional research is needed to produce evidence of the long-term outcomes of SEd programs.

Certain aspects of research findings were unclear or were not mentioned in the methods or results sections, limiting the reliability of the reported findings. Sample composition issues included high attrition rates not being broken down by the individual study intervention or control conditions (Collins et al., 1998). Sometimes sample demographic characteristics were unclear or not described (Manthey et al., 2012b; Yahaya et al., 2010), including studies that lacked a description of the number of sample members with SMI versus other conditions (Morrison et al., 2010; Thompson, 2013). Other studies failed to describe the number of respondents in the control group (Nuechterlein et al., 2008a) or the overarching demographics of the sample, especially in relation to how they could affect outcomes (Manthey et al., 2012b).

Outcome measures around educational engagement or attainment were often not reported separately and instead were reported in conjunction with employment outcomes (Rinaldi et al., 2010). Schindler & Sauerwald (2013) provide some insight into the possibility that educational outcomes alone may have failed to reach statistical significance, but without individual findings reported, it is impossible to understand fully these results. Sometimes key outcomes of interest were not adequately explained or measured even within the context of relatively rigorous designs. For example, Rinaldi, Perkins, & colleagues (2010) included an extensive follow-up data collection in their study--assessments at four time points, extending to 24 months post baseline. The primary outcome of interest in this study was the number of clients working or studying (without an operationalized definition of "studying"). This study missed the opportunity to capture explicit educational enrollment and attainment information. In another study, reported outcomes in the article text were confusing and not well-justified using sound methodological procedures (Yahaya et al., 2010). Fortunately, there is an emerging small body of research studies examining the impact of SEd programs using RCTs (Nuechterlein et al., 2008a; Smith-Osborne, 2012a); however, most comparative outcomes between treatment and control groups for educational and employment have not yet been published.

Adaptations to traditional SEd programs are an important step toward fully understanding the benefits of SEd programs and their services. Some examples of adaptations to traditional SEd programs are cognitive remediation (Kidd et al., 2012a; Kidd et al., 2012b; Kidd et al., 2014), understanding how SEd services are being provided at community agencies (Morrison et al., 2010), and SEd programs serving veterans (Smith-Osborne, 2012a) or Orthodox Jewish communities (Shor & Avihod, 2011). These early adaptations largely serve to provide information about how the program and its services are adapted to meet the specific needs of a target population or community. At this point, there is little generalizable outcome data about how these adaptations in SEd programming or services provide long-term benefits to these target populations.

3.6.2. Unanswered Research Questions

Findings from the literature review pointed to several remaining unanswered research questions. These questions are summarized in the following bullets:

  • SEd/SE approaches have shown that participants have higher levels of "school activity," but this is not the same as demonstrating higher rates of degree attainment and, even more importantly, a change in life status as a result of these advanced degrees (improved standing in the labor market) (Mueser & Cook, 2012). Because so many studies are short-term and/or focus on only course completion, it is difficult to draw conclusions about impact on degree completion, job acquisition (as a result of new degree status), and ultimate employment. So, two critical largely unanswered research questions still remain:

    • Do SEd programs enable individuals to complete a course of study that successfully leads to increased educational attainment as represented by a post-secondary degree or certificate?
    • Can SEd programs enable individuals to successfully get and sustain jobs?

       

  • How can services offered within SEd programs be tailored to best address individual functioning, skills, needs, preferences, and age cohort (Leonard & Bruer, 2007)? Are different SEd model variations needed for various client profiles (e.g., GED support versus vocational training versus traditional 4-year college)?

  • What is the ideal participant profile for participation in a SEd approach? As discussed by Leonard & Bruer (2007), two particularly well-known and controlled studies examined included, as part of their selection criteria, students with above-average intelligence and students with a mean education of at least 13 years (Hoffmann & Mastrianni, 1993; Unger et al., 1991). Are SEd programs too challenging for clients whose language, reasoning, or other academic skills are inadequate? Should clients with poorer educational backgrounds be offered a separate type of training program more focused on adult basic education to build skills?

  • How should SEd programs be best integrated with existing evidence-based practice models? For example, how can SEd and SE be woven together to best meet the needs of individuals with SMI? Are they both part of a service continuum? Which clients would benefit most from one approach over the other (or both)? Are some clients best suited to immediately receive SE services, whereas others could benefit from moving through a SEd model and then to SE?

  • How do SEd programs recruit and engage participants?

  • How do existing SEd adaptations need to be tailored to better address the needs of secondary and post-secondary students, both for those with long-standing psychiatric conditions or with first-episode psychosis?

  • A cadre of recent trials has examined SEd/SE approaches with individuals experiencing first psychotic episodes. Would these integrated SEd/SE interventions be equally or more effective among individuals with trauma, mood disorders, or other symptom profiles?

3.6.3. Methodological Limitations

There are several methodological limitations to studies within the published SEd literature. These limitations hinder opportunities to better understand the impact of SEd programs on key outcomes of interest. Some methodological limitations include the following:

  • Trials needs to be designed with follow-up data collection that extends 3 or more years from baseline to adequately capture longer-term educational attainment and job sustainability outcomes. Most SEd studies are limited by 1-2-year follow-ups (or less), which is an insufficient amount of time for most individuals to complete a full degree requirement.

  • Larger sample sizes in SEd outcome studies are needed to analyze differences in outcomes by demographic characteristics and mental illness/symptomology.

  • Research studies should more commonly use analytic techniques to account for high rates of attrition that occur in the context of the research study to minimize outcome biases.

  • Experimental research needs to match comparison and intervention participants on key characteristics and level of functioning to really understand what works best for whom.

  • Studies of SEd program replication are lacking (across program developers, sites, or communities). Without this type of evaluation and implementation research, it is difficult to know how easily transportable various SEd approaches may be.

3.6.4. Other Gaps in Knowledge that Prevent Supported Education Program Dissemination and Scale-Up

Methodological limitations not only weaken the SEd evidence base, but they also limit the possibility for broader SEd program dissemination. We identified two primary gaps in the SEd knowledge base that impede larger-scale-up of SEd programs.

  • More information is needed on the ideal service context for SEd interventions. What implementation issues are particularly apparent for one context versus another?

  • Efforts are needed to resolve the tremendous service financing hurdles that many SEd programs in the field face.

3.6.5. Additional Data Necessary to Consider Supported Education an "Evidence-Based Practice"

SAMHSA houses a National Registry of Evidence-based Programs and Practices (NREPP, http://www.nrepp.samhsa.gov/01_landing.aspx). It is one of the leading sources of information on evidence-based practices in substance abuse and mental health. NREPP is a searchable online registry and includes more than 350 interventions to date. NREPP was developed to help the public learn more about evidence-based interventions that are available for implementation. NREPP is a voluntary, self-nominating system in which intervention developers elect to participate. After a nomination is submitted, an independent committee reviews intervention evidence to decide whether it meets certain criteria and rates the methodology of the intervention's supporting evidence. NREPP publishes a report called an intervention summary on its web site for every intervention it reviews.

In considering what might be necessary for SEd programs to be considered "evidence-based practices," it is helpful to consider the NREPP program requirements. To apply to receive an NREPP review, an intervention must meet the following minimum requirements:

  1. The intervention has produced one or more positive behavioral outcomes (p<0.05) in mental health or substance abuse among individuals, communities, or populations. Evidence of the positive behavioral outcome(s) has been demonstrated in at least one study using an experimental or quasi-experimental design. Experimental designs include random assignment of participants, a control or comparison group in addition to the intervention group, and pre/post-test assessments. Quasi-experimental designs include a control or comparison group and pre/post-test assessments but do not use random assignment. Studies with single group, pre/post-test designs do not meet this requirement. Significant differences among groups over time must be demonstrated for each outcome.

  2. Implementation materials, training and support resources, and quality assurance procedures have been developed and are ready for public use (SAMHSA, 2011).

  3. The results of these studies have been published in a peer-reviewed journal or other professional publication (e.g., a book volume) or documented in a comprehensive evaluation report. Information must be included in publications to enable independent ratings of six research quality indicators. Each indicator is given a rating of 0 (total absence of evidence, not acceptable), 2 (some evidence, moderate acceptability), or 4 (acceptable):

    • Reliability of Measures: Outcome measures should have acceptable reliability to be interpretable. Here, "acceptable" means reliability at a level that is conventionally accepted by experts in the field.
    • Validity of Measures: Outcome measures should have acceptable validity to be interpretable. Here, "acceptable" means validity at a level that is conventionally accepted by experts in the field.
    • Intervention Fidelity: The "experimental" intervention implemented in a study should have fidelity to the intervention proposed by the applicant. Instruments that have tested acceptable psychometric properties (e.g., inter-rater reliability, validity as shown by positive association with outcomes) provide the highest level of evidence.
    • Missing Data and Attrition: Study results can be biased by participant attrition and other forms of missing data. Statistical methods as supported by theory and research can be employed to control for missing data and attrition that would bias results, but studies with no attrition or missing data needing adjustment provide the strongest evidence that results are not biased.
    • Potential Confounding Variables: Often, variables other than the intervention may account for the reported outcomes. The degree to which confounds are accounted for affects the strength of causal inference.
    • Appropriateness of Analysis: Appropriate analysis is necessary to make an inference that an intervention caused reported outcomes.

       

By these criteria, only a handful of studies would be eligible for an NREPP review nomination. Only six separate interventions were tested using an experimental or quasi-experimental design (including a comparison group): Collins et al. (1998), Hoffman & Mastrianni (1993), Gutman et al. (2009), Kidd, Kaur et al. (2014), Killackey, Jackson, & McGorry (2008), Nuechterlein, Subotnik, Turner et al. (2008a), Nuechterlein, Subotnik, Ventura et al. (2008b), and Smith-Osborne (2012a, 2012b). Because trials are ongoing, three of these interventions do not provide sufficient evidence at this time. The two oldest studies did not find sufficient evidence of a positive behavioral impact and lacked key information on implementation. The most promising candidate intervention is the one tested by Killackey, Jackson, & McGorry (2008) that examined SE with integrated SEd components. The SEd aspects of this intervention approach are not well-described in the two publications available; consequently, it is hard to judge the degree to which this approach moves beyond a traditional SE intervention. By this analysis, we would consider SEd programs as a promising practice. This is consistent with SAMHSA materials developed about SEd program approaches (see https://store.samhsa.gov/shin/content/SMA11-4654CD-ROM/BuildingYourProgram-SEd.pdf). Unfortunately, there is not currently a SEd program tested with sufficient rigor and including sufficient evidence of behavioral change to be nominated for consideration as an evidence-based practice.

Evidence-based practice status for SEd is hampered by study design and lack of positive behavioral outcomes. For the SEd program approach to move from a promising to evidence-based practice, a long-term demonstration project is needed. One particular promising SEd model will need to be tested in a way comparable to the Mowbray trial, but without the methodological flaws and including a longer-term follow-up period. Cook & colleagues' (2005a) multisite RCT of SE should be seen as a model. With the development of the SAMHSA (2011)/University of Kansas SEd fidelity scale, there is the opportunity to quantify the degree to which individual programs are abiding by principles seen as core to SEd approaches. Future trials can now include the fidelity scale as a way of understanding how variation in program fidelity affects client outcomes. This should speed up the process of information that will be necessary for SEd programs to be established as "evidence-based."

View full report

Preview
Download

"smiSEd.pdf" (pdf, 891.42Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®