Assessing the Field of Post-Adoption Services: Family Needs, Program Models, and Evaluation Issues. Evaluation Issues. 3.3 Program Context Issues


PAS evaluations are hampered by evolving program models, complex service packages, and small service populations.

Several characteristics of the way in which PAS programs are developed and administered also affect their evaluation. The field of PAS is young, and program models continue to evolve, so that evaluation must focus on a moving target. Because PAS programs serve entire families, whose members have diverse needs, they typically consist of a package of coordinated interventions rather than a single, more readily evaluated intervention. PAS programs, like adoption, may be administered at the state, county, or community level, creating problems of adequate sample size to detect outcomes. Finally, there is a notable lack of demand for evaluation of PAS programs from the agencies that fund them.

The rapid and recent growth of PAS programs means that there have been limited opportunities for program maturation and stability. Among the PAS programs described in this project's literature review (Barth, Gibbs, and Siebenaler, 2001), few were more than 10 years old. The field has developed rapidly in that time, spurred by both the recent acceleration in adoptions and the influx of federal funds. This relatively fast-paced environment has allowed few opportunities for service delivery models to be refined, outcomes to be tracked, or findings to be shared across sites. Evolving program models, while enriched by new information, can wreak havoc on evaluation if program objectives, participants, or interventions are redefined in midcourse. The dearth of evaluations to date means that newer programs have little shared knowledge to build on, forcing their staff to reinvent the evaluation wheel.

In considering evaluation, a distinction must be made between PAS interventions chap151; a clearly defined set of services delivered to families with similar needs chap151; and PAS programs, which bring together an array of interventions with different objectives and activities to serve a broad range of adoptive families. PAS interventions may operate in the context of a broader PAS program (as does Illinois's Adoption/Guardianship Preservation Program) or independently, as in the case of Portland, Oregon's, PAFT program. Examples of PAS programs with multiple interventions include the statewide programs in Georgia, Massachusetts, and Virginia.

Comprehensive services are needed to meet adoptive families' needs but are difficult to evaluate.

The distinction between intervention and program has important ramifications for evaluation. Interventions with specific populations, activities, and outcomes are far more amenable to systematic evaluation. In fact, the majority of published evaluations, such as Illinois's Adoption/Guardianship Preservation Program, PAFT, and PARTNERS, are of specific interventions. PAS programs that do not structure data collection so that families can be grouped by services received will have difficulty identifying outcomes from their work. An example is Casey Family Services, in which data collected on a comprehensive PAS program could not support links between outcomes and services received. The comprehensive nature of PAS programs in many cases is integral to their design, and there is much to be learned, of course, in examining these programs. Yet this does make evaluation more challenging. Descriptive evaluations that monitor which kinds of families use different mixes of services, and how families move among different services over time, offer valuable lessons for ongoing program development. Outcome evaluations of such programs (rather than their component interventions) may not, however, be sufficiently informative to justify the resources they require.

There is, nevertheless, a middle ground between intervention-specific and comprehensive evaluations that the evaluation of PAS programs can pursue. Programs like Multi-Systemic Therapy (MST) and Assertive Community Treatment (ACT) are flexible and open-ended but have enough structure to be evaluable. Indeed, these programs also work with high-risk populations of children and young adults and have shown their efficacy under very demanding experimental conditions.

Like other child welfare services, adoption and post-adoption services are administered at either the state or local level. PAS programs that are developed at the local or regional level will encounter several barriers to evaluation. Among states responding to the Illinois State University (ILSU) survey, approximately one-half reported that PAS were delivered statewide. As noted earlier, programs that serve relatively small numbers of families will have difficulty discerning patterns in service utilization and demonstrating outcomes. Evaluation expertise is also less likely to be available to local programs than at the statewide level. In addition, the start-up cost of evaluation design will be proportionally more burdensome for a small program. Statewide models, in which a single program model is delivered statewide (as in Oregon) or regionally (as in Massachusetts and Texas), are far more amenable to evaluation.

Statewide PAS programs are more likely than regional ones to have adequate funds and service populations for evaluation.

A final barrier to evaluation among PAS programs is the apparent lack of demand from funding agencies. Among the case-study states, evaluation was generally included in the request for proposals. However, there was little indication that program sponsors are setting clear standards for evaluation or actively advocating for stronger evaluations. Given the natural focus on service delivery among program coordinators, it is unlikely that they will go beyond what is required of them in evaluation. Among state adoption managers interviewed, none cited any pressure to document the activities or effectiveness of the PAS programs they fund.

More basically, funders are not requiring that programs be evidence-based, building on rigorously evaluated work with troubled children and families. Although families routinely indicate that PAS must be adoption sensitive, they need not be developed entirely anew for this population. Bringing science to PAS calls for reconsidering the development of rigorous PAS methods, as well as evaluations.

Funding agencies may soon demand data to document PAS program benefits.

It may be that PAS programs are currently being funded based on the high visibility of foster care adoptions and the common sense appeal of supporting adoptive families. The testimony of one adoptive parent may be far more persuasive in a legislative committee than the best possible evaluation data. However, higher standards on accountability for requested funding are likely at some point in the future, particularly as many states face budget shortages. The field need look no farther than the relatively meteoric rise and fall of intensive family preservation services to understand that family testimony and anecdote do not help a field reach its potential. Strong theory- and evidence-based interventions that are adapted to adoption and rigorously tested are the best strategy for ensuring the future of PAS.

View full report


"report.pdf" (pdf, 399.11Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®