|Some evaluation barriers are related to the clients, structure, and context of PAS programs.|
Limited statistical power. Adoptive families are relatively few in number, and not all adoptive families require PAS, so the total number of families served may be fairly small. PAS programs will thus be limited in the extent to which they can describe patterns of needs and services for specific subgroups, and they will have difficulty demonstrating statistically significant differences in service use or outcomes. Compounding the problem of small numbers is the fact that outcomes achieved may be relatively modest. Outcomes such as adoption disruption or dissolution that may be prevented by PAS are relatively rare; however, the pervasive effect of early trauma suggests that they will occur in some families no matter what supportive services are provided. Improvements in problem behavior and family relationships may also be confounded by developmental changes as children move toward adolescence and its typical disturbances.
Diverse goals and services. The client-driven approach typical of PAS programs creates several limitations to evaluation. First, variations in services received make data on satisfaction or other outcomes more difficult to interpret. Second, the outcomes of interest will vary according to family needs. Evaluators must choose between tailoring outcome measures to the specific issues of the family (that is, having greater specificity but smaller groups) and measuring outcomes more broadly (increasing statistical power but with less informative measures). Third, because families use the service on an as needed basis, discontinuing and reentering as their concerns change, it is difficult to identify points at which pre- and post-measures should be administered. Finally, in family-focused programs, evaluators must choose between collecting data from all family members, which increases respondent burden and may obscure outcomes, and limiting measurement to family members with the most acute needs, which raises concerns about stigmatizing problem children.
Limited program data. To date, program data have been of little help in evaluating PAS programs. PAS programs may not be incorporated into Statewide Automated Child Welfare Information System (SACWIS) data because services are contracted out, or because of concerns about confidentiality. Data about services are often limited by the lack of standard nomenclature for characterizing PAS components. Administrative data systems are further handicapped by the lack of data on adoption subsidies, which are used by families to purchase services they need. Yet subsidy data are often not linked with case records, and are not always captured in a longitudinal format that allows capture of subsidy history and changes.
Rapid program evolution. The rapid and recent growth of PAS programs has allowed limited opportunities for program maturation and stability. Among the PAS programs described in the literature review and case studies, nearly all were less than 10 years old, allowing few opportunities for service delivery models to be refined, outcomes to be tracked, or findings to be shared across sites. Evolving program models can wreak havoc on evaluation if program objectives, participants, or interventions are redefined in mid-course. Newer programs have little shared knowledge to build on, forcing their staff to reinvent the evaluation wheel.
Interventions vs. programs. In considering evaluation, the distinction between PAS interventions clearly defined sets of services delivered to families with similar needs and PAS programs arrays of interventions with different objectives and activities to serve a broad range of adoptive families has important ramifications for evaluation. Interventions with specific populations, activities, and outcomes are far more amenable to systematic evaluation, and the majority of published evaluations are of specific interventions. PAS programs that do not structure data collection so that families can be grouped by services received will have difficulty identifying outcomes from their work, although descriptions of children and families served and services used over time may offer valuable lessons for ongoing program development. Outcome evaluations of such programs (rather than their component interventions) may not, however, be sufficiently informative to justify the resources they require. Programs like Multi-Systemic Therapy (MST) and Assertive Community Treatment (ACT) may offer a middle ground, being flexible and open-ended but with enough structure to be evaluable.
State vs. local programs. Like other child welfare services, adoption and post-adoption services are administered at either the state or local level. PAS programs that are developed at the local or regional level will encounter several barriers to evaluation. These include small numbers of families served, with corresponding loss of statistical power; limited access to evaluation expertise; and proportionally higher burden of evaluation start-up costs. Statewide models, in which a single program model is delivered statewide or regionally, are far more amenable to evaluation.
Lack of demand from funding agencies. A final barrier to evaluation among PAS programs is the apparent lack of demand from funding agencies. Among the case-study states, there was little indication that program sponsors are setting clear standards for evaluation or are actively advocating for stronger evaluations. More basically, funders are not requiring that programs be evidence-based, building on rigorously evaluated work with troubled children and families. It may be that PAS programs are currently being funded based on the high visibility of foster care adoptions and the common sense appeal of supporting adoptive families. However, higher standards of accountability for requested funding are likely at some point in the future, particularly as many states face budget shortages. The relatively meteoric rise and fall of intensive family preservation services demonstrated that family testimony and anecdote do not help a field reach its potential. Strong theory- and evidence-based interventions that are adapted to adoption and rigorously tested are the best strategy for ensuring the future of PAS.