Feasibility Study for Demonstration of Supported Education to Promote Educational Attainment and Employment among Individuals with Serious Mental Illness: Final Report. 6. Synthesis: Supported Education Needs and Opportunities


6.1. Current Supported Education Program Context

A confluence of contemporary policy and practice make this investigation of the feasibility of a SEd demonstration fortuitous. Both the recent WIOA and the early intervention for SMI set-aside in the SAMHSA Block Grants described earlier provide new opportunities for funding for SEd to key populations. This is in addition to existing funding streams and related policies that can be accessed for this service. These include, for example, special education, VR, Medicaid waivers, and SE. Opportunities for SEd in terms of funding and policy are complemented by the increased need for, and experimentation with, SEd practice. The urgency of the need for SEd programs is also seen among institutions of higher education. These institutions have a burgeoning student population with mental health conditions, and college counseling centers are swaying under the weight. Retention and graduation rates for these students are particularly poor. In terms of practice and model development, SEd program development and evaluation have recently received increased attention, especially for individuals with first-episode psychosis, as can be seen in the ongoing NIMH-funded RAISE study. This confluence sets the stage for a feasible SEd demonstration project and indicates that SEd is on the cusp of widespread and sustained implementation, given the opportunity to be tested as an evidence-based practice.

The previous chapters provide guidance on how to capitalize on this fortuitous occasion for SEd. The following synthesis presents project findings on model development, model development needs, funding, funding needs, evaluation and research, evaluation and research needs, and the feasibility of a future SEd demonstration project.

6.2. Model Development

The literature review, environmental scan, and site visits shed light on principal issues concerning development of a model of service to support the educational goals of individuals living with mental health conditions. This section presents findings shared across these activities. Findings include recognizing that the variability among SEd program models is largely due to differences in service context. Despite differences, a shared set of core components is present across SEd efforts. Findings show that SEd is often integrated and delivered in tandem with SE services, but this integration can be beneficial and disadvantageous. Finally, post-secondary campus settings can offer unique opportunities, distinct from traditional SEd services, to support students with mental health conditions in a college environment.

6.2.1. Program Variability

Great variability exists across programs and services that provide education supports to individuals with mental health conditions. Much of the variability stems from the service setting, which can range from specialty mental health settings (e.g., hospitals, clubhouses, community mental health centers) to primary and post-secondary education settings and to state VR agencies. A specific target population (e.g., veterans, first-episode psychosis, transition-age youth) can also dictate how a program is structured and delivered. Variability can also be attributed to the shifting of available financial and staffing resources and to SEd efforts being modified as needed to address real-time needs of individuals working toward educational goals.

6.2.2. Core Components/Goal Consensus

Even with considerable variability across specific SEd programs and efforts, there appears to be consensus on the critical components of the service. As highlighted in the literature review (Chapter 3), (Waghorn, Still, Chant, & Whiteford, 2004) identified ten core features of SEd programs:

  1. Service coordination with professionals outside of the SEd program.

  2. Specialized career counseling, including vocational planning and exploration.

  3. Specialized, program-trained staff with time allocated explicitly to SEd programs.

  4. Financial assistance.

  5. Skill building to facilitate integration into the academic environment, including stress and time management and academic or study skills training.

  6. On-campus information about student rights and resources.

  7. On-campus or off-campus mentoring and support, individual or group support, or peer support.

  8. Coordination with post-secondary education institutions to facilitate course access or within-course assistance.

  9. Access to tutoring, library assistance, and other forms of supplemental educational support.

  10. General support (off-campus support preferred) for the multiple individual barriers and life stressors that can lead to educational attrition.

These same core features, or slight variations thereof, were also noted as key SEd components in the environmental scan and the site visits. Although the specifics may vary depending on program setting (e.g., mental health vs. campus), common components included specialized staff with a dedicated effort to SEd, counseling for careers and educational goals, facilitating financial aid, skill building for educational success, facilitating educational enrollment and retention including acquiring educational accommodations, information about rights and resources, mental health support, coordination with post-secondary education institutions, accessing supplemental educational supports, and providing general supports regarding other noneducation-specific barriers and life stressors. All SEd programs and efforts provided some combination of the aforementioned components.

Some features stood out and were consistently noted and valued, specifically, the presence of dedicated staff, who had supporting educational goals as part of their work and who were committed to helping individuals with mental health conditions meet these goals. This commitment to the work was identified as equal to, if not more important than, a staff person's professional discipline or level of education. Also shared is the understanding that these components need to exist within an environment--be it campus-based or a mental health care setting--that supports mental health and recovery and is dedicated to being free from stigma.

Although no singular standard exists for measuring SEd participant outcomes and tracking success, SEd efforts consistently reported similar goals for participants across program settings. These goals included individuals having an identified educational goal (preferably student led) and individuals enrolling in relevant classes, accruing course credits, and attaining certificates or degrees. Most programs identified an ultimate goal as better employment opportunities, higher income, and lessened dependence on disability benefits, although these distal outcomes could not be measured. These shared SEd components, combined with these shared education outcomes, suggest a common conceptual framework that unifies programs and initiatives that support the educational goals of individuals with serious mental health conditions.

6.2.3. Increasing Supported Education/Individual Placement and Support Integration with Supported Employment

The integration of SEd and SE models--specifically IPS--was frequently seen in SEd programs targeting young adult populations with psychotic or related disorders. This integration is considered particularly appropriate for individuals experiencing first-episode psychosis, because educational experiences are often interrupted by illness onset. Moreover, many individuals with psychiatric conditions need to work and go to school simultaneously, suggesting the need for integrated SE and SEd services. However, as with SEd alone, no singular model integrates SEd and SE. Furthermore, some environmental scan respondents expressed concern that SEd receives fewer resources and attention when combined with IPS, especially when IPS fidelity, rather than education outcomes, is tied to service reimbursement.

6.2.4. Supported Education/Post-Secondary Education Integration Success

Partnerships between SEd programs and post-secondary education settings were key to supporting individuals in their educational goals. SEd programs emphasized the importance of outreach and involvement on college campuses, not only to educate faculty, staff, and students about available SEd resources, but also for SEd staff to fully understand campus services and processes and to create relationships with key campus-based stakeholders (e.g., offices of disability services, counseling, and health). The University of Minnesota provided an exciting example of how to create a campus-wide culture supportive of mental health and wellness, while simultaneously tackling stigma about mental illness from top administrative levels (e.g., Provost's Committee on Student Mental Health) and on-the-ground advocacy efforts (e.g., Active Minds chapters). Campus initiatives are challenged, however, by limited resources dedicated to supporting student mental health.

6.2.5. Supported Education Model Development Needs and Opportunities

This synthesis suggests the following needs for the development of a fully specified, replicable, and testable model for SEd.

  • Specifying SEd Core Components: Specification around the core components of SEd should be increased. This could include matching specific components to SEd activities and to measureable outcomes. A first step could include examining existing program-specific SEd manuals and various SEd efforts being implemented across the country to further operationalize components and activities.

  • Identifying SEd Staffing Requirements: Staffing requirements should be further elucidated to reflect the range of education, disciplines, and training that contribute to skilled SEd staff. This should include an emphasis on specified skill sets and the capacity to support individuals with educational goals. It should also include developing and routinizing training supports coupled with ongoing coaching and mentoring.

  • Defining SEd Specialist Tasks: The role of an education specialist needs to be clearly defined, not only specific to a standalone SEd program, but also when integrated with SE. This includes defining discrete tasks and activities associated with supporting educational goals, while also emphasizing inter-personal and relational skills that facilitate the strong relationships that are the foundation of the work between a SEd participant and a SEd specialist.

  • Operationalizing SED/IPS SE Integration: Strategies on how to integrate SEd and IPS SE need to be further defined and operationalized. This should include defining measurable goals and outcomes specific to educational goals and milestones, as well as strategies for staff on how to balance and integrate education and employment goals.

  • Operationalizing Campus Best Practices Supporting Student Mental Health: A set of best practice guidelines should be developed to highlight successful strategies for improving campus-based supports for students with mental health conditions. Specifics should include how to secure administrative and leadership buy-in and how to partner with key campus departments (e.g., disability services), as well as more ancillary departments (e.g., travel abroad) to address student mental health. Additionally, strategies to normalize mental illness and decrease stigma on campus should be considered.

6.3. Funding

Funding challenges to support SEd program services was a common theme across the literature review, environmental scan, and site visits. Environmental scan participants, in particular, described multiple funding streams used to support SEd service components without one clear, central funding strategy. In the absence of a core funding strategy, SEd programs relied on different funding vehicles that varied in terms of their stability and ultimate sustainability. Some specific funding issues are described in more detail as follows.

6.3.1. Braided Funding

Ultimately, the most feasible funding model for SEd programs will likely be braiding funding from a variety of sources (municipal, federal, state, collegiate, and/or private corporations). Environmental scan respondents hypothesized about this possibility. For example, public special education services can fund education supports for individuals with psychiatric and other disabilities up to 21 years of age but can abruptly end thereafter. Some environmental scan respondents noted the availability of VR dollars for tuition and books. However, VR is not designed to provide the ongoing and sometimes intensive support needs of people with serious mental health conditions. Complementary funding strategies are needed to fill in such funding gaps. Funding from campus disability services offices (for those enrolled in post-secondary education) or Medicaid may be better suited to complement the limitations of VR or special education services.

6.3.2. Medicaid

The research literature supports that education is intrinsically a part of rehabilitation. Educational attainment is necessary to achieve maximum positive occupational outcomes. Medicaid supports rehabilitation services; consequently, many environmental scan and site visit respondents noted that their programs bill Medicaid for SEd program services whenever possible. Many respondents noted the difference between the availability of an SE Medicaid billing code and the lack of such a code for SEd services. Also, programs with joint SE/SEd services described billing specialists' time under the SE billing code. It is interesting that none of the environmental scan or site visit participants noted the opportunity for states to use the 1915(i) Home and Community-Based Services plan option to fund SEd services. If a state amendment is approved and if individuals meet state-defined need criteria, this plan option could offer an opportunity to fund long-term services.

During the site visit, leadership within the EASA program in Oregon specifically described their efforts to expand the use of Medicaid funding for SEd services in the context of IPS. An Oregon state statute mandates an SE billing code. This billing code is specifically tied to use of and fidelity to IPS. Program leaders indicated that Oregon is in the process of creating a modifier for the SE Medicaid billing code. This modifier would allow for SEd activities that are part of SE and IPS to be billed accordingly. This effort represents one example of how states might consider Medicaid support opportunities for SEd activities.

6.3.3. Vocational Rehabilitation Support

VR state agencies were seen as important partners to SEd efforts; this is partly because of their high federal match rate, their ability to fund tuition and books, and new WIOA legislation. However, VR funding is not intended to provide the longer-term educational services and supports often needed by individuals with psychiatric disabilities to truly succeed in attaining an educational goal. Further, VR can fund education activities only when these activities are explicitly directed at facilitating employment. VR funding for SEd services would have to be supplemented by other funding sources that would fulfill the ongoing needs for skill training and support.

6.3.4. Supported Education Funding Needs and Opportunities

This section describes strategies for identifying opportunities that can sustain funding for SEd programming.

  • Braided Funding Case Studies: Those working in the SEd field need to better understand how various programs across the country have and are currently braiding funding to support their SEd program activities. Published case studies that demonstrate successful braided funding strategies in support of SEd services could be widely used to help program administrators circumvent the funding challenges noted in stakeholder discussions across this project.

  • Medicaid Billing Code: The availability of an SE Medicaid billing code has helped to disseminate and sustain SE approaches for individuals with psychiatric disabilities across the United States. A similar Medicaid billing option could support and extend the availability of SEd services to complement employment supports. The availability of this type of billing option would directly benefit young adults with mental illness who are highly likely to have both education and employment goals.

  • Guidelines for SE/SEd Medicaid Billing: Programs described using the SE Medicaid billing code to support the activities of SE/SEd specialists' time; however, procedures for billing joint SE/SEd program activities vary. SE/SEd program administrators could benefit from guidelines that describe how to bill SEd activities that occur as part of IPS or other SE services.

  • Increased Clarity around Medicaid Waiver Option Processes: Program administrators implementing SEd programs could benefit from enhanced clarity around the availability of Medicaid funding to support education services through the 1915(c) or 1915(i) Home and Community-Based Services plan options. This guidance could come in the forms of a state Medicaid director letter, program guide, frequently asked questions document, or fact sheet.

  • WIOA Expansion: The recent WIOA expansion offers an opportunity for SEd program implementation and support through VR. The expanded emphasis on WIOA to address career needs of 15-21-year-olds will certainly involve supporting their education goals. VR dollars, with their high federal match for state dollars, can incentivize SEd services for this population. There is also an opportunity to braid the dollars associated with WIOA with Medicaid to provide the rehabilitation services that are concomitantly needed.

6.4. Evaluation and Research

Synthesizing across the environmental scan, site visits, and literature review, concordance was found on the readiness of providers to conduct data collection and on the perceived key outcomes of SEd, thus setting the stage for future evaluation efforts. It also became clear that additional data that will be necessary to establish a platform for considering SEd as an evidence-based practice. These issues are described in more detail here.

6.4.1. Data Collection Readiness

The potential feasibility of a SEd program evaluation was evident in the ongoing data collection occurring across many programs included in this project via the environmental scan and site visits. SEd program sites appeared ready and able to support data collection efforts. In fact, many of the SEd programs included in this project were collecting data, sometimes even outside of the requirements of their particular funding source. These current individual SEd program data collection methods and procedures already in practice could be used as a "springboard" for a broader initiative to study SEd program outcomes.

6.4.2. Agreement on Key Outcomes

Across the literature review, environmental scan, and site visits, there was consensus on what outcomes are important to measure in order to assess SEd program impact. Along with capturing data on service utilization and participant characteristics, key agreed-upon outcomes for SEd programs focused on educational attainment as measured by indicators such as course enrollment data, the number of credits completed, and graduation rates.

6.4.3. "Evidence-Based Practice" Status

SAMHSA houses the NREPP (http://www.nrepp.samhsa.gov/01_landing.aspx). It is one of the leading sources of information on evidence-based practices in substance use and mental health treatment. NREPP offers guidelines that are helpful in considering what might be necessary for SEd programs to be considered an "evidence-based practice." These guidelines involve the following basic minimal requirements: (1) demonstration of positive behavioral outcomes in at least one study using an experimental or quasi-experimental design; (2) availability of implementation and quality assurance materials ready for public use; and (3) published results documented in a peer-reviewed or other professional publication. By these criteria, only a handful of studies examining the impact of SEd intervention could even be considered for an NREPP review nomination based on their study design. Also, upon further examination, those experimental or quasi-experimental studies available either have not demonstrated sufficient positive behavioral outcomes or do not have publicly available implementation or quality assurance materials. Currently, no SEd program has been tested with sufficient rigor or includes sufficient evidence of behavioral change to be nominated for consideration as an evidence-based practice.

Progress towards designation as an evidence-based practice status for SEd is hampered because there have been few trials of SEd with comparison groups; trials with comparison groups have not shown sufficiently compelling outcomes of interest. Program enhancements to the SEd approach have been made since the first SEd RCT conducted by Collins et al. (1998); consequently, it may be the case that these new program enhancements will lead to improved education and employment outcomes. There is suggestive evidence that this will be the case. For the SEd program approach to move from a promising to evidence-based practice, a long-term demonstration project is needed. This project would need to use random assignment, measure degree completion, and include other key program impacts such as employment; health, mental health, or recovery; and community participation. The project should be large enough to examine outcomes across various program setting types and special populations of interest.

6.4.4. Evaluation and Research Needs and Opportunities

Following the example of SE, establishing SEd as an evidence-based practice will in turn promote expansion and implementation of the service. Thus, additional evaluation and research are needed for SEd. Foremost is the need for RCTs with sufficient power to enable adequate analysis of SEd outcomes alone and for long-term follow-up data collection efforts to demonstrate ultimate impacts on employment.

Randomized Controlled Trials

Many promising findings highlight the positive impact of SEd programs on youth and young adults with SMI. However, the current state of evidence is not sufficient to support SEd programs as an evidence-based practice. No RCT with sufficient power to identify differences in SEd outcomes for youth and young adults has been conducted and published. This level of rigor is necessary for any future trial of SEd. Moreover, the ideal SEd trial will be powered sufficiently to identify differences in SEd outcomes (education and ultimate employment) for individuals with educational attainment goals. The importance of this was noted within the literature review, in which findings suggested that currently published studies frequently bundle education and employment outcomes together, prohibiting examination of the singular impact of the intervention on education OR employment. Future research and evaluation studies will need to be organized to better understand possible connections between educational/degree attainment, subsequent employment, wage/income, job stability, and ultimately disability status to fully capture the potential impact of SEd programs.

Long-Term Follow-Up

A long-term commitment to tracking key SEd program outcomes is needed. Longer-term follow-ups are absent in the existing literature; many stakeholders participating in the environmental scan mentioned this problem, and sites visited often reported no funding support to continue long-term data collection. Only a handful of research articles included in the literature review included follow-up data collection beyond an immediate post-program assessment. Only seven of the original research study designs collected follow-up data for 8 months to 3 years beyond preliminary participant program enrollment/baseline. Meanwhile, a typical post-secondary degree/certificate program takes 2-4 years to complete. With the exception of one study with a follow-up data collection period of up to 3 years, none of the follow-up data collection periods in existing SEd trials would have been positioned to gather information on post-secondary degree completion or program certification attainment for program participants.

Taken together, the following specific research and evaluation needs were identified:

  • Rigorous Evaluation and Research Designs: SEd programs demonstrate a strong ability to support evaluation studies and data collection efforts; however, existing evaluation efforts are not systematic. Rigorous evaluation and research designs are needed that capitalize on the existing SEd program infrastructure and data collection readiness.

  • RCTs: Rigorous research designed to understand the impact of SEd on core outcomes of interest is needed. In particular, a well-designed RCT could help establish the evidence base necessary to move SEd from a "promising" to an "evidence-based" practice.

  • Follow-Up Data Collection for 3-5 Years (minimum): Any future SEd research or evaluation trial must be designed with follow-up data collection that extends a minimum of 3 years and ideally 5 or more years from baseline to adequately capture longer-term educational degree attainment and ultimately job sustainability outcomes. Most SEd studies are limited by 1-2-year follow-ups (or less), which is an insufficient amount of time for most individuals to complete a full degree requirement.

  • Large Sample Size: Larger sample sizes in SEd outcome studies are needed to analyze differences in outcomes by demographic characteristics and mental illness/symptomology. Larger sample sizes are also needed to allow sufficient power to disentangle the additional benefit of SEd to IPS approaches, separate from their impact on employment outcomes. This would not be feasible in a multisite design.

6.5. Need for and Feasibility of a Future Supported Education Demonstration Project

Results from our literature review, environmental scan, and site visits clearly suggest that a demonstration of SEd is needed. One goal of this study was also to determine the feasibility of such a SEd demonstration project. Project results also indicated that the SEd field would be ready to support a demonstration project. The state of the practice indicates that such a demonstration would need to have two sequential and progressive stages: (1) refine existing fidelity measures, interventions, implementation tools or guides, and develop a demonstration project design; and (2) launch a multisite RCT demonstration project.

6.5.1. Stage 1: Refine Existing Fidelity Measures, Interventions, Implementation Tools or Guides, and Develop a Demonstration Project Design

Because of the existing variability in SEd implementation, some procedures and measures would need to be specified and refined before launching an RCT. Several design details would also need to be resolved. Stage 1 would take approximately 6-12 months to complete.

Test Existing Fidelity Measures

At least two fidelity measures exist for SEd: the University of Kansas Supported Education Fidelity Scale (Manthey et al., 2012a) and the Supported Employment/Supported Education Fidelity Scale for Young Adults with Mental Health Challenges (Frounfelker, Bond, Fraser, Fagan, & Clark, 2014). One measure will need to be selected and/or revised based on its specificity and match with core SEd program goals and practices and the intended RCT intervention. For example, the Frounfelker et al. (2014) scale would be well suited for an SE/SEd demonstration project, whereas the Manthey et al. (2012a) scale would work best within a demonstration project focused exclusively on SEd. In this stage, broader testing of the fidelity scale is needed to establish psychometric properties, validity, sensitivity, and ability to assess changes in program variation over time.

Standardized Supported Education Program Intervention

Given the variability observed in SEd programs, a standard set of activities needs to be established. There are many existing SEd manuals, which can be culled for reproducible procedures and used to standardize the core service delivery components across sites.

Develop Implementation Tools or Guides

Implementation tools and guides are needed, particularly for establishing protocols for training staff, recruiting the target population, and establishing connections with campuses. The SAMHSA SEd promising practice toolkit provides an excellent foundation for this step.

Develop a Demonstration Project Design

Several demonstration project design parameters need to be considered before launching an RCT (Stage 2). These include, but are not limited to, making literature-informed decisions about an expected program effect size (affecting sample size and site capacity recommendations), finalizing a study design (type of control group, decisions about site structure and variability), and giving attention to participant recruitment, retention, and expected attrition over time, as well as within the intervention and control groups. These design development decisions could be easily folded into Stage 1 activities.

  • Expected SEd Program Effect Size: Decisions about statistical power and sample size parameters along with recommendations about the number of demonstration sites will be informed by estimates of an expected program effect size. Effect size estimates are necessary for power calculations. Underpowered studies will not have a good chance of finding a statistically significant difference between a treatment and comparison group (even if it exists). An anticipated effect size can be informed by the SEd literature. However, if adequate detail in the literature does not exist (as might be the case with SEd-specific interventions), an effect size may be estimated from expert discussions around the smallest effect size deemed meaningful to test the impact of SEd programs. Once an effect size is determined, decisions about sample sizes and the number of demonstration sites needed to achieve this sample can be decided.

  • Type of Control/Comparison Group: A comparison group is critical to the next stage of SEd program research and evaluation. Questions around SEd program impact still remain after decades of smaller-scale, nonexperimental trials. These program impact questions can be definitively addressed only within the context of a well-designed RCT. The most probable comparator for a SEd demonstration project is a "treatment-as-usual" condition (rather than a no-treatment control group). An alternative could be a comparison group with comparable attention from a provider that does not deliver SEd services, or perhaps an "active" control that provides minimal SEd services such as informational fact sheets. The choice of a comparison group will affect the degree to which treatment differences are detected between the SEd intervention group and the comparator. Thus, demonstration project design decisions will ultimately affect recommendations around sample size as well as site quantity, so they need careful consideration.

  • Site Structure: Study design consideration will need to be given to a recommendation around the desired state agency structure for administering SEd services within the demonstration project. Study design recommendations will also need to address any required cross-agency partnerships considered necessary for demonstration project SEd service administration. For example, the home for SEd has historically been state mental health agencies and community psychiatric rehabilitation providers; this would be a feasible administering structure for SEd. However, as noted in the report, state agencies of VR can provide important funding for tuition and books and a high federal match for services. Also, each state has a network of local VR offices and counselors that can be accessed for statewide implementation of SEd. A demonstration project design team could consider these issues.

  • Site Variability: This project found that SEd programs can be administered in a variety of settings. One important design development decision will be to determine the degree of interest in understanding how SEd program effects vary by setting or type of site (e.g., campus-based vs. specialty mental health based). More variability and heterogeneity across sites will lead to the need for a higher number of sites and larger demonstration project sample size.

  • Participant Recruitment, Retention, and Attrition: Environmental scan participants and the published literature note challenges related to recruiting and retaining the participation of individuals with SMI in SEd programs. These challenges have direct implications for a SEd demonstration project. For example, attrition should not be anticipated to occur at random. Those program participants who are at highest risk for poor outcomes are especially likely to drop out of an intervention or control group. Attrition may also be of particularly high concern among individuals within the control condition where participants may have less service contact or engagement over time. High, nonrandom attrition affects the demonstration project's power to detect treatment differences and creates biases in project data. Special attention (e.g., participant incentive plans, tracing procedures, engagement with young adult consumers to review proposed study procedures design and instrumentation) should be paid to these issues at the study design phase to maximize the demonstration project's power and minimize the potential for biased data.

6.5.2. Stage 2: Launch a Multisite Randomized Controlled Trial Demonstration Project

Stage 2 provides the basis for establishing SEd as an evidence-based practice through an RCT. This trial would include a multisite design with all sites required to adhere to SEd consensus goals and core components. The program model selected would ideally not include an integrated IPS/SEd program (because a few trials of various integrated approaches are under way). Rather, the greatest field need is to explicitly test the impact of SEd programs rigorously to understand the unique impact of this type of program component (separate from an emphasis on employment supports).

It is important that the demonstration project involve an experimental design, including random assignment with a control group, to best position SEd for consideration as an evidence-based practice. As in the SE trial (Cook et al., 2005a), the control condition need not be a "no-treatment" control; instead, a treatment-as-usual model would be highly encouraged.

The process evaluation would use the tools constructed in Stage 1 to assess program fidelity and implementation activities. The outcome evaluation structure could include short-term, mid-term, and long-term goals and assessments. The demonstration project must evaluate outcomes beyond the 3-year mark--to not do this runs the risk of SEd programs being deemed ineffective because core outcomes of interest have not been allowed sufficient time to develop. Similarly, the trial must include a sufficient sample to be statistically powered to detect program impacts on either employment or educational goal achievement (outcomes), measured independently.

The outcome evaluation would track key service utilization and participant characteristics. Importantly, the outcome evaluation should also include those key outcomes for SEd programs noted across the literature review, environmental scan, and site visits. These include a particular focus on educational attainment as measured by course enrollment data, the number of credits completed, and graduation rates.

Stage 2 would require an additional 3-5 or more years (depending on follow-up length). Many options would support such a process--Stage 2 could proceed conditional upon the completion of Stage 1, or the two stages could be supported simultaneously. It is not uncommon for early multisite trials to involve a design phase in which program fidelity models and training procedures are refined first, then program enrollment begins 12-18 months after this early design phase is completed. A process similar to this, for example, was followed for the recent RAISE trial (Kane et al., 2015).

Designing a multisite SEd study would be comparable with Cook & colleagues' trial of SE (Cook et al., 2005b). In this trial, each site was permitted some variation in implementation, although prior standards were well set. Despite program variability, "fidelity" was conceptualized across diverse programs, and common outcomes were agreed-upon during the trial process. All programs were required to use the same measures, and data were submitted to a central repository. Given the noted variability in SEd programs, this approach would make the most sense. Results of the environmental scan and case studies, in particular, illustrate how helpful (and necessary) natural variation is. Programs will need to be allowed to vary but be held to uniform standards, goals, and components. Standardization and replication of SEd across communities can be achieved by developing tools in Phase 1 that support the implementation of core SEd program goals and components. This, again, would be very comparable with the trial of SE (Cook et al., 2005b).

6.6. Summary

Findings from the current review of research, policy, and practice indicate that SEd is on the cusp of widespread and sustained implementation. A synthesis of the literature review, site visits, and environmental scan suggests that although settings vary widely, there are also common core practices of SEd. Creative braiding of funding will likely be the solution to the absence of a clear funding stream, and guidance on how to accomplish this will aid provider organizations. Taken together, data suggest that a demonstration trial of SEd is both needed and feasible. Existing research and evaluations of SEd programs lack sufficient rigor, adequate sample sizes, and long-term follow-up assessments to produce the platform necessary to demonstrate SEd program impact. Furthermore, provider organizations are well poised to conduct systematic data collection on SEd processes and outcomes. However, to surpass the limitations of the current SEd research described in this report, a two-stage demonstration program is needed: Stage 1 to prepare fidelity and implementation guides and Stage 2 to conduct a multisite RCT with long-term follow-up. Such a program would provide the platform necessary to generate the potential evidence needed to move SEd from a promising practice to an evidence-based practice, thus encouraging future funding and widespread adoption.

View full report


"smiSEd.pdf" (pdf, 891.42Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®