COMPREHENSIVE SERVICE INTEGRATION PROGRAMS FOR AT-RISK YOUTH

By Martha R. Burt, Gary Resnick and Nancy Matheson

The Urban Institute

December 7, 1992

For the Office of the Assistant Secretary of Planning and Evaluation

U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES

---

EXECUTIVE SUMMARY

Estimates suggest that as many as half of today's youth run a moderate to high risk of experiencing school failures or participating in early sexual activity, alcohol and drug use, and criminal behaviors. This report examines programs targeting at-risk younger adolescents, aged 10 to 15.

The results of many years of program impact evaluations demonstrate that single-focus programs targeting at-risk adolescents may not be the most effective way to help youth. Increasing attention is being paid to programs capable of dealing with the whole child, including the child's parents and neighborhood. Thus, a major focus of this report is to learn more about the ability of programs to provide more comprehensive services to youth through service integration. It also examines the barriers and successes programs encounter while attempting to do so.

Project Objectives

The objectives of this project are to:

• Document how comprehensive, integrated services are delivered to at-risk youth between the ages of 10 and 15.
• Identify effective methods of providing comprehensive, integrated services for this population.
• Identify barriers to providing comprehensive services, and means of facilitating service integration for at-risk youth.
• Examine the role of Federal, state, and local government and the nonprofit sector in impeding or facilitating service integration for at-risk youth.
• Examine the extent to which simple lack of services, or insufficient service capacity, is implicated as a barrier, in comparison with eligibility, regulatory, jurisdictional, and other factors.
• Identify issues for further research on the provision of comprehensive, integrated services for at-risk youth.

Methods

To meet these objectives we reviewed the literature concerning the meaning of risk, the prevalence of risk behaviors among youth, and the successes and limitations of traditional programs serving youth. We conducted a literature review and examined issues related to evaluating programs for at-risk youth, and then conducted site visits to nine programs in six locations.

Report Outline

Chapter 1 introduces the study and its objectives. Chapter 2 contains a review of the literature on youth at risk. It examines definitions and prevalence of risk, describes current approaches to service delivery, explores the definition of and motivation behind service integration (SI), and examines the barriers programs face to implementing SI. Chapter 3 delineates the evaluation issues concerning programs serving youth at risk. Chapter 4 introduces the objectives of the site visits and describes the procedures used. Chapter 5 describes each of the nine programs visited. Chapter 6 identifies and discusses issues that cut across programs. Chapter 7 presents a summary of the project and implications of the findings.

Risk Definitions, Risk Prevalence, and Service Integration Issues (Chapter 2)

Some of the key issues discussed in detail in Chapter 2 include the definition of risk used during the project, the difficulties of measuring prevalence of risk, the limitations of traditional services that focus on single problems rather than on meeting the needs of the whole individual, our definition of service integration (SI), the barriers to service integration, and the crucial steps that should be taken to plan and implement SI.

The Meaning of Risk

The conceptual framework for defining risk was developed for this report and consists of four components:

• Risk antecedents: Those environmental forces that have a negative impact on the developing individual by producing an increased vulnerability to future problems in the family, school, or community. Based on our review of the literature, there appear to be three critical risk antecedents for early adolescents; poverty, neighborhood environment, and family environment.
• Risk markers: These are visible indicators of behavior, in public records. Previous research suggests a consistent relationship between these behaviors and risk antecedents, and a well-defined link with increased vulnerability and the onset of potentially negative behavior. We have selected two indicators that are consistently identified as markers for all problem behaviors of adolescence: poor school performance and involvement with child protective services, including out-of-home placement in the foster care system. These two have particular policy relevance because they can be observed in the records of public systems, and allow program planners to target the youth at greatest risk.
• Problem behaviors: These are defined as activities that have the potential to hurt youth, the community, or both. Research has identified these behaviors as those most likely to occur in youth who, earlier, displayed risk markers, or who were living under risk antecedent conditions. We have chosen those behaviors that have most consistently been identified in the literature as signaling potentially more serious consequences for youth in the future, including: early initiation and practice of sexual behavior, truancy or absenting from school, running away from home (or from an out-of-home placement), early use of tobacco, alcohol, and other drugs, and associating with delinquent peers.
• Risk outcomes: These are clearly injurious conditions that have negative consequences for a youth's future development as a responsible, self-sufficient adult. The risk outcomes of primary concern include teenage pregnancy/ parenthood, homelessness, involvement in prostitution, alcoholism and/or drug abuse, delinquency and criminal behavior, school dropout, AIDS, chlamydia and other sexually-transmitted diseases, physical and sexual abuse, and various morbidity and mortality conditions (hepatitis, tuberculosis, pneumonia, accidents, suicide, homicide).

Prevalence of Risk

This report examines the prevalence of the above four components based on the secondary literature. Despite the apparent overlap in risk antecedents and markers, it is difficult to develop a composite estimate of the degree to which adolescents run a high, moderate, or low risk for engaging in problem behaviors or experiencing risk outcomes. Using a simple population estimate based on poverty or neighborhood is very rough and will overestimate the number of youth who go on to experience risk outcomes. The more precision one desires in an estimate of risk, the more difficult the task becomes. This is because antecedents and markers are never perfect predictors, and the quality of data gets significantly worse as the variables are more closely connected to problem behaviors or risk outcomes. Some investigators address this problem by taking the presence of the problem behaviors themselves as their "risk" indicators. But this approach merely begs the question, since the youths it selects as high risk have already done the things one is interested in predicting. A better solution is to use several antecedent and marker variables to predict risk; in general, the more relevant the variables included, the more precise prediction one achieves.

Limitations of Traditional Services

Our survey of traditional services for at-risk youth shows that they often address only a single risk marker or outcome such as adolescent pregnancy, substance abuse, or school failure. This single-problem focus has several limitations. First, such programs usually focus on problems (rather than individuals as a whole) and tend to offer short-term interventions. Programs that try to solve problems quickly and then close the case are not geared toward preventive interventions and often have little staying power. Thus, they do not always address the most pressing needs of their clients. Second, it is sometimes difficult to get other community agencies to fill in the gaps when such single-issue programs cannot meet client needs with their own program resources. Given these problems, programs have tried to increase the comprehensiveness of their own offerings and use service integration to increase their clients' access to a wide range of services offered by other programs and agencies.

Definition of Service Integration

By "service integration" (SI), we refer to procedures and structures that help several service agencies coordinate their efforts to address the full range of service needs presented by youth and families in an efficient and holistic manner. It is important to understand that, despite the common tendency to refer to "comprehensive service integration programs," the terms "comprehensive" and "service integration" are not synonymous. Service integration is merely one method of obtaining comprehensive service coverage, but SI does not guarantee comprehensive service coverage. Furthermore, programs may use different combinations of comprehensiveness and SI. For instance, a mental health program may have arrangements with other agencies to provide additional services, but these services may still be related to mental health rather than to other aspects of client need such as housing or education (this is an example of SI without comprehensiveness). Or, a program may try to make its own service offerings comprehensive rather than relying on outside agencies to fill in service gaps (this is an example of comprehensiveness without SI).

Few existing systems meet all the elements of the SI model with which this inquiry began. Several key elements in this initial idea of SI efforts for at-risk youth are:

• An approach to helping at-risk youth that sees each youth for himself or herself, and also sees the youth as part of a family, neighborhood, and community that may in turn be influenced to reduce the risk that a youth will participate in problem behaviors or experience risk outcomes.
• A comprehensive, individualized assessment at or near the point of intake, that is conducted for each youth and family, to identify the full range of his or her individual and family service needs.
• A coordinated service plan that, based on the needs identified, is developed to ensure that all needs are addressed in an efficient fashion by the program(s) best suited for the task.
• Institutionalized interagency linkages that ensure that service referrals result in actual service delivery. This may entail an interagency case management function, co-location of services at a single site, and/or sharing of other resources among programs.
• Follow-up on service referrals, to ensure that services are delivered in an appropriate manner and that the program coordination structures are functioning effectively.

Barriers to Service Integration

SI efforts face many barriers, including professional training and orientation, administrative procedures, eligibility rules, and the categorical nature of funding. Service agency staff are typically trained in rather narrow, specialized traditions such as mental health or criminal justice services, and may not feel comfortable dealing with other issues or working within an interagency framework. Bureaucratic procedures often obstruct SI efforts because agencies may insist on following their own intake and case processing procedures, and confidentiality requirements may limit their ability to share information about clients with an SI team.

Categorical public and private funding also perpetuates single-issue programs. As long as legislatures and funders structure programs to address specific problem areas, single-issue programs will continue to have difficulty making their services available to populations not specified by their mandate.

Steps to Planning and Implementing SI

Some of the crucial steps that should be considered when planning and implementing service integration include:

• Defining Goals and Objectives. Encourage long-term commitment to the integration effort by creating an independent interagency advisory group to help minimize turf battles and forge a common purpose for service integration partners.
• Identifying the Target Population. Unless the target population is clear, it will not be obvious what services should be incorporated into the effort. There is no definitive profile of youth or families who need SI. However, prime candidates may include families who need help in supportive parenting due to involvement in alcohol or drug treatment or with child welfare because of reports of abuse or neglect. Equally important are families who have none of these problems but who struggle to raise their children with little money and few resources in neighborhoods that pose a constant threat to their children's future.
• Identifying the Services to Be Offered. A comprehensive approach requires considerable variety in the breadth and depth of services available and flexibility in service delivery. The type of services to be offered should be determined on the basis of local needs and resources.
• Mechanisms for Service Delivery. Services may be coordinated in different ways. Clients may have a service agency contact with whom they maintain an ongoing, supportive relationship. When this contact person functions more as a mentor, counselor, or group worker than as a case manager, this individual needs access to someone who can arrange needed services and follow up on referrals.
• Locating the Service Site. Integrated services can be delivered through school-based or school-linked sites, in community sites, through mobile arrangements, or by home visits. The location of an SI effort most likely depends on which agency or group has a committed and dynamic person willing to take the lead in developing and running the program. The site's acceptance in the community is also an important factor. Debates about the appropriate balance of services between on-site and off-site locations center around the relative benefits of ease of access versus teaching clients to negotiate the system themselves.
• Eliminating Administrative Barriers to SI. Agencies participating in SI should have institutionalized linkages that establish the mechanisms for sharing resources. These mechanisms may include co-locating in a single facility, sharing staff financial resources and/or information, and agreeing to provide services to referred people.
• Hiring Staff. Staff should be selected very carefully. Criteria should include their: ability to establish trusting, respectful relationships with youth and families; ability to span professional boundaries to address clients' needs; willingness to support the SI model; and demonstrated sensitivity to issues of racial, ethnic, and gender diversity.
• Creating Flexible Funding. For SI to work best, funding should be flexible, avoiding the rigidities of circumscribed service delivery and the eligibility difficulties associated with categorical funding. Federal and state funding sources should be redesigned to blend together funds from multiple sources that historically have rigid categorical boundaries, to provide adequate and coherent funding for service programs that address multiple areas of need. One promising approach to increasing SI among already functioning programs is using limited new funding to support core SI functions. This effort could be matched by diverting some existing funds to support additional SI efforts and using other existing funds to support regular service delivery--an approach now being used in Kentucky.
• Designing and Using Evaluations Effectively. Evaluators must have extensive early collaboration with program personnel so the measures used are meaningful and cooperation with the evaluation is high. Impact information should be tied to youth and family outcomes rather than simply services delivered. Programs that look good as demonstrations are often diluted upon replication, suggesting that evaluation results are not reviewed in enough detail to assure that critical aspects of programs actually appear in replication.
• Institutionalizing Change. A major goal of SI is to change service delivery systems in a permanent way. But this often does not happen; changes rarely survive the tenure of the key people involved in SI efforts. For SI efforts to produce true system change, the interagency linkages and ways of interacting must be codified into a new approach to standard operating procedure.

Evaluation Issues and Lessons Learned (Chapter 3)

Chapter 3 documents evaluation issues specific to youth-serving programs, to service integration efforts, and to the types of sites selected for this study. Issues examined include defining the participant and the unit of analysis, measuring client risk levels and including them in analyses to understand program impacts, documentation of service delivery, non-client outcomes of interest, differentiating the impact of SI from that of comprehensiveness, evaluation readiness, identifying realistic outcomes to measure, identifying appropriate comparison groups, and reducing attrition to follow-up. The chapter discusses:

• The need to adjust evaluation designs to reflect major elements of program activity, including prevention.
• The importance of assessing client risk levels, documenting service delivery, and including appropriate indicators in a multivariate impact analysis.
• Incorporating measures of the extent of SI, comprehensiveness, program design change, and system change as important aspects of evaluation design for the types of programs examined here.
• The importance of using a strong evaluation design (probably quasi-experimental), having adequate instrumentation to measure key concepts, and obtaining follow-up data from a very high proportion of entry cohorts and comparison group members.

Some of the lessons learned from years of program evaluations are as follows:

• Evaluators should be outsiders rather than program staff, but these individuals need to take the time to get to know the program and work carefully with program staff to develop mutually agreeable arrangements.
• The impacts that programs care most about, such as youth development or leadership training, may be the most difficult to measure adequately.
• An exclusive focus on outcomes and impacts does not always accurately capture the full picture. Quantitative outcomes should be augmented by using qualitative and observational methods to learn not just whether a program "works," but how it works, under what conditions, and for whom. Not knowing these specifics about program-client fit makes it harder to recommend future applications of a demonstration program or to translate results into broader policy directions.

Site Visit Objectives and Procedures (Chapter 4)

The site visits were conducted:

• To understand the full range of program configurations and options for 10- to 15-year-olds, including the programs' sense of their mission or purpose.
• To understand the reasons behind these programs' choices among certain program design alternatives (e.g., whether to emphasize "activities" or "services;" whether to concentrate on prevention or on treatment; whether to adopt a focus on youth, on youth plus their families, on families in general, or on the total neighborhood; whether to strive for comprehensive service delivery).
• To understand the relationship of these programs to their larger community, including both the program's role in the service delivery network and network of supports for youth, and the program's role in relation to other community institutions such as churches and community centers.
• To learn what programs believe are the benefits of a more comprehensive range of services, and what they believe are the benefits and drawbacks of service integration through collaborative arrangements with other agencies.
• To gain a sense of the readiness and willingness of programs of this type to participate in evaluations, and what types of evaluations they might be open to (or have already been involved in).

Site Selection Criteria

We looked for programs that serve clients between the ages of 10 and 15; conduct comprehensive, individualized needs assessments for individual youth; use these needs assessments as the basis for service planning or case management; have developed formal, institutionalized interagency linkages; and conduct standard follow-ups with agencies to which referrals are made to ensure accountability. Not all programs ultimately visited met all five criteria.

Programs Visited

The nine programs in six locations that were ultimately selected represent a mix of program type, geographic location, and racial/ethnic groups served. They include the Belafonte-Tacolcy Center in Miami, Florida; Big Brothers/Big Sisters of Greater Miami in Coral Gables, Florida; Chins Up Youth Care Homes in Colorado Springs, Colorado; Garfield Youth Services in Garfield County, Colorado; I Have a Future and Oasis Center in Nashville, Tennessee; Sunset Park-Center for Family Life in Brooklyn, New York; Teen Connections in the Bronx, New York; and Communities in Schools in Houston, Texas. These nine programs can be characterized as follows:

• Age Range: Between 50 to 100 percent of clients served by these programs are age 10 to 15.
• Gender: One program serves only girls; the remainder serve both boys and girls, but tend to have more boys.
• Race/Ethnicity: Two programs serve almost entirely African-American youth, two serve mostly white youth, one serves mostly Hispanic youth, two serve a mixed group of Hispanic and African-American clients, and two have very ethnically mixed groups of users.
• Focus of Activities/Services: Three programs focus their efforts mostly or exclusively on the youth themselves, but may assist a youth's family if it becomes apparent that help is needed; three programs focus on youth in some of their activities and place a heavy emphasis on involving the families of youth in other components of the program (e.g., for "caseload" clients); three programs have some activities mainly for youth, some services that involve youth and their families, some offerings for any interested community member, and an overarching goal of changing and empowering the whole community.
• Program Model: The nine programs include one mentoring program, one program focusing on a geographically defined community, one operating almost entirely in the schools, three operating in both schools and the community, and three that are community-based. Five of the programs use case management and three offer crisis-oriented, short-term services.

Program Descriptions (Chapter 5)

The nine programs visited are each described using the following categories: brief history; current mission, goals, and objectives; service configuration; current clientele/users; type and makeup of SI network; funding sources; and evaluation.

Cross-Program Issues (Chapter 6)

Chapter 6 summarizes the findings from site visits with respect to the following issues:

• Clarity about who is and who is not a client;
• Client risk levels and their implications for program service offerings and for evaluation;
• Program orientation toward strengthening families and/or neighborhoods;
• The cultural context of program operation;
• Scope and variety of service delivery, and the meaning of comprehensiveness as programs see it;
• Service integration issues, including the scope and variety of networks and SI arrangements, history and evolution of SI, perceived impacts, and difficulties encountered and ways of handling them;
• Program choice and tradeoffs with respect to client age range, prevention/treatment orientation, activities/services orientation, youth-family-community orientation; and
• Evaluation issues, including program interest in and perceived pay-offs from evaluation, past history of evaluation activities, level of documentation currently available, and our perceptions of the feasibility of a multi-program evaluation with programs such as these nine we visited.

Below we briefly summarize the site visit findings concerning service integration and evaluation issues.

Service Integration

The site visits confirmed our initial view that programs use a variety of configurations to facilitate access to services and augment service delivery to program clients. Programs use both formal and informal arrangements. For instance, in the formal category, some programs: have staff from other agencies come to deliver a service either on a permanent or a scheduled basis, contract with other community agencies to provide services or join a multi-agency team that meets regularly to handle clients who need services from several agencies, and have contracts to provide services to clients of other programs on the site of the other program. In the informal category, some programs rely on consciously worked out relationships between program caseworkers and referral agencies to improve clients' chances of getting needed services. These informal links are important because they are more common than formal SI arrangements. But because such networks often break down when key staff leave, they are no substitute in the long term for formal commitments. Finally, programs use different types of volunteers to expand their service options. Mentoring was the most common volunteer activity, though one program we visited provided emergency shelter through host homes with volunteer families and two arranged with volunteer members of the business community to provide services.

Information Sharing.

We found that most of the programs with formal SI linkages (four of the nine programs visited participate in formal SI efforts) have worked out arrangements for release of information as needed, usually on a case-by-case basis. For instance, some programs release information temporarily for the purpose of having a multi-agency team design a client plan, but the releases are not general and do not go beyond the framework of that specific plan.

On the other hand, programs that rely on informal cross-program service delivery mechanisms report that information sharing is a continuing problem. Even when they are trying to get help for a specific client, they say they cannot name the client so as not to violate the client's privacy. Also, because of the informality of arrangements, information that needs to be shared is not shared because no feedback mechanism exists to assure that a referring person ever learns what happened with a referral.

• Perceived SI Benefits. The programs with well-functioning SI arrangements cite several benefits, including: clients receive both an increased number of services and more appropriate services, participating agencies follow through on their commitments with greater speed and thoroughness, youth are much less likely to fall through the cracks, and staffing patterns stabilize because the programs' community-building philosophy is attractive to staff and increases their commitment to the program.
• Difficulties Encountered. All of the programs we visited try in various ways to live with the disadvantages of categorical funding described above. The greatest difficulties cited other than those inherent in the current system of service funding are related to "turf" issues--between agencies, between program staff and staff of an agency with which they want to work, and between ethnic groups. Between-agency issues include several agencies competing for the same dollars to develop similar programs; different agencies with control over some of the same youths not agreeing over the best approach, and therefore not willing to commit resources to the case(s); and different agencies having different goals for the program.

Agency-to-program and program-to-agency tensions included disciplinary differences in approach engendering hostility or distrust, a key person in an agency being threatened by a program person's expertise, and insensitivities with regard to peak workload periods.

One program cites ethnic tensions in their larger service community over whether agencies affiliated with and serving particular ethnic groups would get their own resources or would have to be under the control of agencies affiliated with different ethnic groups.

A final problem cited involves interactions within an agency on SI. Programs that have tried SI without commitment from both agency directors and line workers have run into difficulties.

Conceptualizing SI More Broadly

This project began with a view of service integration that is client-driven. It assumes that an agency has clients with service needs that it cannot meet entirely with its own resources, and that it becomes involved in formal interagency linkages to access services for its clients. We have learned that this view of SI is quite narrow and formal. It does not encompass several of the situations found during site visits, which appear to the researchers to epitomize an ideal of SI as service development and community coordination. Several programs we visited make themselves available to develop services as their need is manifested in the community. If the program itself or other agencies with youth-serving responsibilities identify major unmet needs, the community of agencies can negotiate exactly what is needed, who can best provide it, how the various agencies in town will relate to the new service, and other similar issues. These agencies serve as mortar for their community networks--they hold them together, fill in the gaps, and facilitate smooth service delivery whether through their own services or the services of other agencies. They may do relatively little through formal or even informal referrals of their clients to other agencies, yet they help create a truly integrated service delivery system.

Evaluation Issues

Two key conditions determine the readiness or "evaluability" of a program for an outcome evaluation: willingness and capability. Willingness refers to a program's interest in evaluation and perception that evaluation can help advance program goals. Capability refers to a program's current resources for evaluation--the skills of its staff, its data collection capabilities, and its commitment of staff time to evaluation activities. An important factor affecting both these conditions is the program's history of participating in evaluation efforts, since this experience will contribute to staff attitudes about the experience and to existing capabilities. Among the nine programs visited, those with the highest levels of capability are usually those with more positive attitudes toward evaluation. But in some programs the two dimensions do not exactly correspond.

In terms of willingness, most of the programs show an interest in doing more evaluation research and, in particular, assessing program outcomes. Many directors specifically indicate that they want to do some form of longitudinal follow-up of their clients as an indicator of their program's success. The enthusiasm of some programs relates to earlier positive experiences with evaluation studies. Among programs that appear more hesitant about evaluation research, one cites a bad experience it had with the evaluator for the national demonstration program of which it was a part, and one cites its concern about an evaluation's ability to reflect the complexity of client experiences in the program. This program thinks that the rather cut-and-dried approach to outcome assessment used by one evaluation did not do justice to either their services or the benefits their clients derived from the program.

In terms of capability, the programs visited can be grouped into low, moderate, and high capability. Low-capability programs lack the existing resources required for an evaluation study, including staff knowledgeable about evaluation research, sophisticated information systems, and a central unit or department responsible for putting information together. These programs have some trouble tracking the involvement of outside agencies, which is an important component of documentation for SI types of programs.

Moderate-capability programs maintain some form of a computerized database system into which service and client statistics are entered regularly. Some of these programs still rely on the executive director to analyze the service statistics, but generally top management is supported by volunteers and staff who complete the forms and do the initial tabulation of the information. Moderate-capability programs have the ability to use the documented information for the purposes of planning and internal evaluation. Such programs sometimes have quite specific plans for improving their evaluation capability. While these programs have strong interest in evaluation, some feel resources available for evaluations are insufficient.

High-capability programs have highly sophisticated management information systems and staff specifically assigned to do the data entry, compilation, and summary statistics. They usually are conducting or have conducted some form of evaluation. They possess a high level of readiness, even though some have had some negative experiences with prior evaluation research. All of these programs clearly indicated that any costs incurred by doing evaluation research were more than compensated for by the benefits of the information obtained.

The high-capability programs appear the most ready, and with some additional resources the moderate-capability programs may also be helped to participate in a multi-program evaluation. However, at least two issues need to be addressed in order to design an outcome evaluation that includes some or all of the sites visited and that will identify the effects of comprehensive service provision and SI models.

• The first issue concerns the choice of comparison or control groups. It is not clear from the site visits how all programs may be able to identify a potential group of non-intervention clients. Although community-based controls may be formed, there are the risks of contamination with the program participants and the possibility of these youth and families entering the programs themselves at some point. One potential avenue for the choice of control or comparison group is to draw these individuals from a matched sample living in an adjacent community that does not have such a program.
• The second issue is how to resolve the variability in information currently documented by programs, particularly client risk level information and service provision characteristics that would be amenable to classification. It would also be important to identify a standard minimum data set that all sites provide for the cross-site analysis.

All programs involved in a cross-site evaluation should become involved in this decision-making process so they develop ownership and positive attitudes toward a cross-site evaluation. Given the special features of these programs, the measures should not just assess individual changes, but should also identify the effects of the programs on the community and on the interagency service delivery network.

Summary and Implications (Chapter 7)

The literature on at-risk youth and programs that serve them indicates that a comprehensive approach has the best chance of helping youth avoid negative behaviors and outcomes. SI is one way to increase the comprehensiveness of program offerings by facilitating access to services available in the community that a program does not itself provide. The programs we visited are all complex, all strive for a high degree of comprehensiveness, and most are involved to some degree in service integration to achieve it.

To us, the most striking implication of this project's findings is the need to conceptualize service integration more broadly. We think it is important to recognize the efforts that some programs make to develop their community's capacity to serve youth, by identifying and working to develop services to address unmet needs. Also exciting was learning about Joint Initiatives in Colorado Springs--a service integration effort with the highest level of commitment from all relevant local agencies.

Another important finding of this project is that these agencies have a very strong interest in conducting evaluation research, but most do not have the resources to go beyond the usual program statistics to assess outcomes or the role of particular services in achieving those outcomes. Almost all programs are interested in participating in outcome evaluations if they have adequate resources and technical assistance. They also feel it is important that any evaluation design reflect the complexity of their program activities and the many ways that youths, families, and community members may participate in them.