Although most adoptions have positive outcomes for the children and their families, many families need supportive services during some part of their child's development. In response to these needs, many states have developed post-adoption service (PAS) programs and other supports for adoptive families. The U.S. Department of Health and Human Services contracted with RTI International to examine these rapidly growing and evolving programs. Research questions covered the need for PAS, characteristics of existing programs, and strategies used to assess program effectiveness. RTI, in collaboration with the University of North Carolina at Chapel Hill School of Social Work, conducted a literature review, case studies of five PAS programs, analysis of secondary data, and an assessment of evaluation issues affecting PAS.
Current and Recent Evaluations of PAS Programs
Little research has been completed on PAS programs to date. The study team reviewed five published evaluations and ongoing evaluation activities within the case study sites (Georgia, Massachusetts, Oregon, Texas, and Virginia).
Types of Evaluations Conducted
Needs assessments document the extent of demand for PAS and support service planning, sometimes including information on anticipated needs and severity of need. Most commonly conducted by state agencies via mail surveys of families receiving adoption subsidies, they are therefore unable to determine the exact number of adoptive families needing services. Their findings may not represent the needs of the larger population of adoptive families, especially those who adopted privately or internationally.
Data on the characteristics of children and families served, which many PAS programs collect, can provide useful information for program delivery. The information may include demographics, adoption history, and descriptions of child and family strengths. Data on the amount and types of services delivered are used for both evaluations and program planning. These data may serve as mediating variables in outcome evaluations. Case-study states used a variety of approaches for documenting service delivery, ranging from case records to a web-based case management system.
Client satisfaction assessments collect feedback from adoptive parents to help PAS programs improve service delivery. Case-study states used mail or telephone surveys to assess satisfaction. The validity of client satisfaction data is often limited by poor response rates, but the information is seen as useful by program staff.
Outcome evaluations are not common among PAS programs, due to the difficulty of demonstrating effects, the lack of an identified point at which to measure outcomes, and other challenges. They are essential in documenting program effectiveness and comparing alternative service delivery approaches. There is no consensus yet on which measures or clinical assessment instruments are best suited to the needs of adoptive families and most appropriate for different program models.
Services and Programs Evaluated
Crisis intervention and counseling services were more likely than other interventions to include outcome evaluations, using either subjective ratings by parents or workers, events such as clinical placements, or clinical assessments. Evaluations of information and referral services typically report on characteristics of families and children served, services delivered, and client satisfaction, using telephone or mail questionnaires and event-tracking databases. Given the brief nature of these interactions, evaluations requiring collection of additional information are not common. Evaluations of comprehensive PAS programs tend to be similar to evaluations of crisis intervention programs in terms of data collected and methods used. These evaluations are limited in their ability to identify which services are effective for which families.
Data Collection Methods
Several factors influence the choice of data collection method: the type of evaluation, the PAS offered, the respondent (e.g., program staff or adoptive family), and the case management system (e.g., paper or computerized); as well as the evaluation goals and level of evaluation funding. A web-based system used in one case-study state combined case management functions with compilation of data for evaluation.
EVALUATION CONSTRAINTS AND INCENTIVES
Among evaluation constraints identified in the case-study interviews, several were common to other service delivery arenas. Funding was the most commonly identified constraint, limiting programs' ability to conduct evaluation activities and to contract with external evaluators. With limited funding, program coordinators frequently put service needs ahead of evaluations. Evaluation expertise is related not only to funding but also to the availability of staff with qualifications in evaluation design, data collection, and analysis. Program staff may need to invest time in orienting evaluators to issues that affect the choice of outcome measures, instrument, and timing of data collection.
PAS staff expressed concerns about the impact of evaluation on families. The staff viewed time spent completing evaluation instruments as encroaching on the therapeutic interactions without benefiting the family, and introducing a more clinical tone to the process. Finally, evaluation was seen to be of limited value to programs. Case-study interviews revealed few instances in which PAS coordinators or providers identified ways in which evaluation findings had been, or were expected to be, useful to them.
Barriers Specific to PAS
Some evaluation barriers were specific to PAS programs. Because adoptive families make up a small part of the population and may not need or use PAS, small population sizes may limit evaluations. Many programs will have difficulty describing subgroup patterns or demonstrating statistically significant differences in service use or outcomes. Compounding these problems is the fact that even effective PAS programs are likely to achieve only modest outcomes. Program goals may be stated in terms of prevention of adoption disruption or dissolution. Because such outcomes are rare chap151; and may be inevitable in some instances even with optimal post-adoption support chap151; impact on their prevention is difficult to demonstrate.
Flexible, family-centered service delivery may further limit evaluation by making it difficult to identify consistent points for data collection, to link outcomes to services received, and to choose respondents for outcome measures. Finally, PAS programs generally have limited administrative and program data. States may be unable to track services provided by contractors and the relationship between services provided by PAS programs and those funded with subsidies. In addition, the lack of consistent service classifications makes it difficult to compare across programs.
The ways in which PAS programs typically are developed and administered also affect their evaluation. The field of PAS is young, and program models continue to evolve, so that evaluation must focus on a moving target. Because PAS programs serve entire families, whose members have diverse needs, they generally comprise a package of coordinated interventions rather than a single, more readily evaluated intervention. Like adoption, PAS programs may be administered at the state, county, or community level, creating problems of adequate sample size to detect outcomes. Finally, there is a notable lack of demand for evaluation of PAS programs among the agencies that fund them.
Although the list of barriers is long, some characteristics of PAS programs may actually facilitate evaluation. Those involved with the programs have a genuine curiosity about how programs can be improved. Because of their investment in adoption issues, many adoptive parents are quite open to participating in evaluation. Finally, the field can draw on the experience of other areas of child and family services for evaluation instruments that can be adapted to the needs of adoptive families.
FUTURE DIRECTIONS IN EVALUATING POST-ADOPTION SERVICES
Two general strategies are suggested in this report. First is the development of fundamental evaluation tools that can reduce the start-up costs of evaluation to programs and increase comparability of evaluations across programs. Specific measures include the following:
- Develop consistent service classifications to facilitate program description and analysis.
- Identify a core set of "best practice" models with recommended evaluation strategies.
- Develop a basic program data set to describe child and family characteristics and functioning.
- Improve data on adoption subsidies by capturing significant events and facilitating retrieval and linkages to other data.
- Develop programmed child and family assessments.
- Conduct rigorous evaluations that provide conclusive data on PAS effectiveness.
Second, funding agencies at both the federal and state level can consider strategies to reduce barriers to evaluation. Among the measures they could take are the following:
- Promote evaluation as a tool for program improvement rather than an arbitrary chap151; or nonexistent chap151; requirement.
- Structure evaluation processes so that they are useful to programs and families.
- Earmark funds for evaluation so that is not perceived as impinging on service delivery.
- Fund programs for multiple years to allow adequate time for planning, full implementation, and identification of effects.
- Provide evaluation technical assistance to PAS programs to assist with design, data collection, and analysis.