Beyond these common challenges, some of the fundamental characteristics of PAS programs seem to conspire against assessment of the programs and their outcomes. These include the relatively small number of families served, the family-centered and client-driven nature of service delivery, and the long-term and formidable challenges of adoption.
The hallmarks of PAS programs-services that are tailored to family needs and used as needed over the life of the adoption-create challenges to evaluation.
Adoptive families are relatively few in number, estimated at less than 3 percent of the population (Chandra, Aloma, Maza, and Bachrach, 1999). Among adoptive families, an even smaller proportion has service needs that are adoption specific or are not met by existing community resources. Thus, the total number of families served may be fairly small. This is particularly true for PAS programs that are not delivered statewide, statewide programs in less populous states, or programs that are restricted to specific types of adoptions. The modest scale of these programs does not in any way argue against their importance. However, programs with relatively small populations served will be limited in the extent to which they can describe patterns of needs and services for subgroups, such as families with preschool children or those whose children have a history of multiple preadoptive placements. They will also have difficulty demonstrating statistically significant differences in service use or outcomes.
Compounding the problem of small numbers is the fact that outcomes achieved may be relatively modest and diverse. While program goals may focus on the prevention of adoption disruption or dissolution (2) or out-of-home placement, these outcomes occur in no more than 15 percent of the special needs child population (National Adoption Information Clearinghouse, 2002). Further, the pervasive effect of early trauma suggests that these outcomes will occur in some families no matter what supportive services are provided. Improvements in problem behavior and family relationships may also be confounded by developmental changes as children move toward adolescence and its typical disturbances. Programs need to identify mediating measures that can detect meaningful benefits to the family, such as connectedness that is sustained even when the child cannot live in the family, or parental confidence in their ability to respond to the challenges presented by the child. Yet improvements in parental efficacy or parent-child closeness do not have the compelling impact of disruption or dissolution prevention.
PAS programs are typically flexible in their delivery. The length of service and mix of services provided are tailored to the needs of the family, in accordance with the guiding principles articulated by the National Consortium for Post Legal Adoption Services (NCPLAS) (Howard and Smith, 1997). While this client-driven approach enables programs to respond to families' specific needs, it creates several limitations to evaluation, particularly with respect to outcome evaluation. Because families use the service on an "as needed" basis, discontinuing and reentering as their concerns change, it is difficult to identify points at which pre- and post-measures should be administered. Designs that assess change at a case closure may be flawed because many families do not formally exit, and then may often reappear months or years later. Yet designs that rely on fixed-length measurement chap151; e.g., one year after case opening chap151; may capture families in the midst of treatment, so the changes that are measured may not reflect the eventual apex of improvement. An additional concern is that if families are not in touch with the program at that point, follow-up will require considerable effort and data are likely to be incomplete.
The needs and concerns of adoptive families are diverse, and PAS programs typically tailor services to meet family needs. This creates two challenges to evaluation. First, variations in services received make data on satisfaction or other outcomes more difficult to interpret. Second, the outcomes of interest will vary according to family needs. Evaluators must choose between tailoring outcome measures to the specific issues of the family (that is, having greater specificity but smaller groups) and measuring outcomes more broadly (increasing statistical power but with less informative measures).
PAS programs are typically family focused, also in accordance with the NCPLAS principles. Recognizing that all family members are affected by adoption, services are designed to meet the needs of adoptive parents and siblings, as well as adopted children. This strength of PAS programs again creates difficult choices around data collection. Collecting data from all family members increases respondent burden, and it may obscure outcomes by including the experience of family members without significant concerns or substantial involvement in the program. On the other hand, limiting measurement to those family members with the most acute needs may downplay the systemic nature of adoptive family dynamics and raise concerns about stigmatizing "problem children."
Administrative data have been used for sampling and for examining program outcomes in family preservation (Schuerman, Rzepnicki, and Littell, 1994), child welfare (Wulczyn and Zeidman, 1997), and welfare evaluation projects (Barth, Brown, Cuccaro-Alamin, and Needell, 2002), but have been of little help in evaluating PAS programs. Many of these services are contracted out and not covered by the Statewide Automated Child Welfare Information System (SACWIS). Some jurisdictions with relatively comprehensive SACWIS have been late in developing their adoption modules because of concerns about confidentiality. When data about services are gathered, they are often vague, as there is yet no standard nomenclature for characterizing PAS components.
Administrative data systems are further handicapped by the lack of data on adoption subsidies. Subsidies are a key aspect of service provision because many states expect families to request and use subsidies to purchase the particular services that they need. Yet subsidy data are often not linked with case records and are not always captured in a longitudinal format. In some states, changes in subsidy amounts are not saved and dated; in such cases the history of the subsidy and its uses cannot be captured.