[ Main page of Report | Contents of Report ]
Available research on post-adoption services and supports is largely descriptive and based on only a few of the programs implemented in this rapidly developing field. Two groups of evaluations are discussed: (1) those identified by our literature review and (2) ongoing evaluations in the case-study states. While the former group is generally more fully developed, the latter group represents ongoing and recent efforts by well-respected programs.
Needs assessments. Needs assessments are commonly done, although rarely published for circulation outside the sponsoring state. They describe the kinds of services most needed by families, in terms of recent needs, anticipated needs, priorities, or unmet needs. These studies can be used to document the need for a PAS program and support planning of services to be provided. State-sponsored needs assessments in the field of adoption have generally gathered information from surveys of families who adopted from a state’s public welfare system and were receiving an adoption subsidy. Contacting these families is facilitated by the fact that states keep contact information for subsidy payment purposes.
| Needs assessments and program documentation are more common than outcome evaluations. |
Characteristics of children and families served. Regularly collecting data on the characteristics of children and families served is common practice by case managers providing health and social services, often at intake and assessment. In the case of PAS programs, evaluators have used the data gathered by case managers and program staff as part of a process evaluation, yielding a range of potentially useful information to guide PAS program direction and service delivery. Data on families and children served include basic demographic information, history prior to adoption, risk to adoption, family problems and strengths, and family functioning. Clinical instruments are sometimes used to describe child and family functioning, as well as to provide a baseline for outcome evaluations.
| Needs assessments and program documentation are more common than outcome evaluations. |
Services delivered. The collection of data on service delivery and usage is also fairly common for PAS programs; these data have been used in several PAS evaluations. These data are also critically important to planning PAS programs and funding. Data on services delivered may serve as mediating variables in outcome evaluations, establishing the effect of specific types of services or a threshold service level necessary for effect. If used as part of an outcome evaluation, services need to be documented as they are delivered rather than summarized at case closing. PAS program records are also unlikely to capture services that the family may have received from private providers or other sources not affiliated with the PAS program, which may influence outcomes.
Client satisfaction. An assessment of family satisfaction with services received is a common evaluation approach to improving PAS program staffing and programmatic planning. Most client satisfaction surveys involve the adoptive parent. In several of the case-study states, program staff and evaluators used client satisfaction surveys to guide service delivery. As with needs assessments, the validity of client satisfaction survey data is often limited by poor response rates. Nevertheless, these efforts serve as a useful barometer for program staff and offer an opportunity to maintain communication with adoptive families regarding their needs and preferences.
Outcomes. Outcome evaluations are the least common of all evaluation types. Among challenges inherent in outcome evaluations are the difficulty of demonstrating effects, particularly for less intensive interventions, and the lack of a clear point at which outcomes are to be measured. While outcome evaluations are not necessarily appropriate for all interventions, there is likely to be increasing pressure on PAS programs to document their effectiveness, and increasing interest within the field in comparing alternative service delivery approaches. A variety of measures have been used for outcome assessment, including clinical assessments, changing incidence of events such as adoption disruption or out-of-home placements, goal attainment, or subjective assessments by workers or parents. Child and family clinical assessments offer detailed measures of child and family outcomes for more intensive interventions, with the opportunity for pre/post comparison. There is no consensus yet on which measures are best suited to the needs of adoptive families and most appropriate for different program models.
| Intensive interventions are more amenable to evaluation than comprehensive PAS programs. |
Crisis intervention and counseling. Evaluations of crisis intervention and counseling services are more likely than other interventions to include outcome evaluations, using either subjective ratings by parents or workers, events such as out-of-home placement, or clinical assessments. If services are offered on a flexible rather than time-limited schedule, evaluators must grapple with the problem of defining an endpoint at which outcomes are to be measured. The evaluations reviewed had widely varying study populations, ranging from 22 children and families (Medina) to 1,162 children and families (Illinois). Evaluations with very small populations will lack the statistical power needed to demonstrate significant differences in outcomes.
Information and referral services. Because of the low intensity of these services, evaluations are generally limited to descriptions of the families and children served and service utilization. Measures of client satisfaction may be the most appropriate outcome measure. Given the brief nature of the interaction, evaluations that require collecting additional information from the client (other than that collected within the information and referral request) are unlikely to be feasible. One case-study state, however, used data from its web-based case management system to document the degree to which problems identified in the information and referral service were eventually resolved.
Evaluations of comprehensive PAS programs. Evaluations of comprehensive PAS programs tend to follow the pattern of evaluations of counseling and crisis intervention by compiling data on child and family characteristics, clinical assessments, risks to adoption, service usage, client satisfaction, and case outcomes. Evaluation methods are also similar and include clinical assessments, case records, and parent feedback forms. Evaluations that attempt to assess the entire program rather than specific components will inevitably be limited in their ability to link services to outcomes. The nearly limitless combinations of amount and type of services that may be used, compounded by the diversity of adoptive families, make it difficult to unravel the threads of what services are effective for which families.
Evaluations of PAS interventions and PAS programs have employed a variety of data collection methods. Choice of method is influenced by several factors, including the type of evaluation, type of PAS offered, type of respondent (e.g., program staff or adoptive family), type of case management system (e.g., paper or computerized), evaluation goals, and level of evaluation funding. For example, an evaluation of a support group for adoptive parents might use focus groups with members of the group. A linked computerized case management system could allow for analysis of aggregate data on child and family characteristics, service usage, and case outcomes entered by caseworkers. If ensuring high-quality service delivery is a primary goal, conducting a client satisfaction survey might be an appropriate method. If funding allows, an evaluator could conduct an outcome evaluation using clinical assessments and even a comparison group. Among evaluations reviewed, some general patterns were observed:
[Go to Contents]
Observations on factors influencing PAS evaluation are drawn largely from case-study interviews. Although we cannot assume that they are generalizable, the differences encountered by these well-regarded programs are likely to be present in other states as well.
| As in many programs, a lack of funding and expertise, and interference with program activities are barriers to evaluation. |
Funding. Funding was the evaluation barrier most frequently mentioned by state adoption program managers and PAS coordinators and providers. Evaluation requires substantial resources, and program coordinators frequently place higher priority on meeting service needs than on evaluation. Funding agencies contribute to this situation if they require evaluation without specifying the level at which it is to be done or do not allocate adequate resources for both service delivery and evaluation. Among the case-study states, the one with the most sophisticated evaluation allocated approximately 5 percent of its budget to evaluation, hardly adequate for a new program area in which service delivery models and evaluation methods are not well established.
Evaluation expertise. Contracting with an external evaluator requires a greater commitment of program funds but provides access to a higher level of expertise than is likely to be found among program coordinators or staff. Even if a PAS program is willing to commit the resources to contracting with an external evaluator, however, finding an evaluator with adequate understanding of adoption issues may be difficult. Given the recent development of PAS programs, there is neither a large base of published research nor an extensive network of experienced researchers.
Interference with program activities. PAS program staff were concerned that the time required for evaluation activities added to their workload and impinged on their interactions with families, without necessarily providing any direct benefit to the family. Program staff were also concerned that evaluation activities introduced a clinical tone to their interaction that was at odds with their efforts to normalize the adoption experience, especially when instruments focused on child and family problems.
Limited value to program. PAS coordinators or providers rarely found evaluation findings to be useful in their practice. While data were used to quantify the volume of services delivered or families’ satisfaction with the program, evaluation was not seen as a source of new and useful input on substantive questions of program design. If evaluation findings do not inform program development, staff are less likely to be willing participants in evaluation activities.
| Some evaluation barriers are related to the clients, structure, and context of PAS programs. |
Limited statistical power. Adoptive families are relatively few in number, and not all adoptive families require PAS, so the total number of families served may be fairly small. PAS programs will thus be limited in the extent to which they can describe patterns of needs and services for specific subgroups, and they will have difficulty demonstrating statistically significant differences in service use or outcomes. Compounding the problem of small numbers is the fact that outcomes achieved may be relatively modest. Outcomes such as adoption disruption or dissolution that may be prevented by PAS are relatively rare; however, the pervasive effect of early trauma suggests that they will occur in some families no matter what supportive services are provided. Improvements in problem behavior and family relationships may also be confounded by developmental changes as children move toward adolescence and its typical disturbances.
Diverse goals and services. The client-driven approach typical of PAS programs creates several limitations to evaluation. First, variations in services received make data on satisfaction or other outcomes more difficult to interpret. Second, the outcomes of interest will vary according to family needs. Evaluators must choose between tailoring outcome measures to the specific issues of the family (that is, having greater specificity but smaller groups) and measuring outcomes more broadly (increasing statistical power but with less informative measures). Third, because families use the service on an “as needed” basis, discontinuing and reentering as their concerns change, it is difficult to identify points at which pre- and post-measures should be administered. Finally, in family-focused programs, evaluators must choose between collecting data from all family members, which increases respondent burden and may obscure outcomes, and limiting measurement to family members with the most acute needs, which raises concerns about stigmatizing “problem children.”
Limited program data. To date, program data have been of little help in evaluating PAS programs. PAS programs may not be incorporated into Statewide Automated Child Welfare Information System (SACWIS) data because services are contracted out, or because of concerns about confidentiality. Data about services are often limited by the lack of standard nomenclature for characterizing PAS components. Administrative data systems are further handicapped by the lack of data on adoption subsidies, which are used by families to purchase services they need. Yet subsidy data are often not linked with case records, and are not always captured in a longitudinal format that allows capture of subsidy history and changes.
Rapid program evolution. The rapid and recent growth of PAS programs has allowed limited opportunities for program maturation and stability. Among the PAS programs described in the literature review and case studies, nearly all were less than 10 years old, allowing few opportunities for service delivery models to be refined, outcomes to be tracked, or findings to be shared across sites. Evolving program models can wreak havoc on evaluation if program objectives, participants, or interventions are redefined in mid-course. Newer programs have little shared knowledge to build on, forcing their staff to reinvent the evaluation wheel.
Interventions vs. programs. In considering evaluation, the distinction between PAS interventions — clearly defined sets of services delivered to families with similar needs — and PAS programs — arrays of interventions with different objectives and activities to serve a broad range of adoptive families — has important ramifications for evaluation. Interventions with specific populations, activities, and outcomes are far more amenable to systematic evaluation, and the majority of published evaluations are of specific interventions. PAS programs that do not structure data collection so that families can be grouped by services received will have difficulty identifying outcomes from their work, although descriptions of children and families served and services used over time may offer valuable lessons for ongoing program development. Outcome evaluations of such programs (rather than their component interventions) may not, however, be sufficiently informative to justify the resources they require. Programs like Multi-Systemic Therapy (MST) and Assertive Community Treatment (ACT) may offer a middle ground, being flexible and open-ended but with enough structure to be evaluable.
State vs. local programs. Like other child welfare services, adoption and post-adoption services are administered at either the state or local level. PAS programs that are developed at the local or regional level will encounter several barriers to evaluation. These include small numbers of families served, with corresponding loss of statistical power; limited access to evaluation expertise; and proportionally higher burden of evaluation start-up costs. Statewide models, in which a single program model is delivered statewide or regionally, are far more amenable to evaluation.
Lack of demand from funding agencies. A final barrier to evaluation among PAS programs is the apparent lack of demand from funding agencies. Among the case-study states, there was little indication that program sponsors are setting clear standards for evaluation or are actively advocating for stronger evaluations. More basically, funders are not requiring that programs be evidence-based, building on rigorously evaluated work with troubled children and families. It may be that PAS programs are currently being funded based on the high visibility of foster care adoptions and the common sense appeal of supporting adoptive families. However, higher standards of accountability for requested funding are likely at some point in the future, particularly as many states face budget shortages. The relatively meteoric rise and fall of intensive family preservation services demonstrated that family testimony and anecdote do not help a field reach its potential. Strong theory- and evidence-based interventions that are adapted to adoption and rigorously tested are the best strategy for ensuring the future of PAS.
| PAS program evaluations may be facilitated by staff dedication and lessons learned from other family service programs. |
Although outweighed by barriers to evaluation, some facilitators were noted in case study interviews. First, the rapid evolution of the PAS field has given rise to a ferment of new approaches, and those involved in these programs tend to share a genuine curiosity about “what works?” and “is this an improvement on other approaches?” Evaluations that respond to this appetite for program improvement could garner substantial cooperation in spite of the evaluation barriers described above. Second, adoptive parents (a major source of data for such evaluations) have an enormous investment in adoption-related topics and will often be ready participants in evaluation. Although PAS program staff are appropriately protective of parents’ time and desire to normalize family life, the study team’s experience suggests that many adoptive parents are very willing to participate in activities that can improve PAS programs. Finally, although PAS programs are relatively new, they can draw upon evaluation experiences in other areas of child and family services and on the existence of psychometrically tested instruments for both children and families. Use of these instruments allows considerable streamlining of evaluation design, as well as the opportunity for comparability across evaluations.
[Go to Contents]
A substantial boost in our knowledge of PAS will require many steps, building from two general strategies. First, evaluators should collaborate with PAS leaders to develop fundamental evaluation tools that will reduce the start-up costs for programs and will increase comparability of evaluations across programs. Second, funding agencies can consider strategies to reduce barriers to evaluation by making them more useful to program staff, ensuring that evaluation activities do not impinge upon service delivery, and providing adequate resources in the form of both funds and expertise. This section provides specific recommendations to address each of these agendas.
Develop consistent service classifications. The spectrum of PAS has now been identified in several descriptive efforts (e.g., Smith and Howard, 1997). Yet these have not been carefully described so that different raters would consistently categorize the kind of service received — that is, the difference between classifying a service as therapy, advocacy, or case management may not be readily distinguished. If the field is going to describe post-adoption activities — and eventually link these to case characteristics, consumer satisfaction, and client outcomes — then a more precise nomenclature is needed. In addition, we need basic research to determine the overlap between a variety of services to understand whether these interventions can be separated out (and monitored or tested separately) or combined into clusters of services.
| Basic evaluation tools can reduce evaluation burden and increase data comparability. |
Identify “best practice” models with recommended evaluation strategies. The field of PAS has been dramatically strengthened by the high level of innovation evidenced during the past decade. As discussed earlier, the fast pace of program development has to some extent come at the cost of evaluation. With the emerging recognition of promising models, it should now be possible to propose a core set of interventions with associated evaluation strategies. Such an effort could be led by an expert panel, working in consultation with program coordinators in the field. For each intervention component, the panel would characterize intended participants, objectives, program activities, process and outcome measures and recommended instruments. While not all programs would follow the strategies, identification of recommended measures and instruments could considerably reduce evaluation design costs and facilitate cross-site comparisons.
Develop a model data set. Program data are rarely used to evaluate PAS, although most programs collect information on family characteristics and services provided. A model data set, offered to PAS providers in a basic database format, would facilitate consistent data collection across programs and jurisdictions, and speed the development of a broad understanding of who uses PAS and how. A contractor and Technical Work Group should determine whether this basic data collection format would also include measures of services provided, child or family functioning, or satisfaction with services.
Improve data on adoption subsidies. Improved administrative data about subsidies could provide a variety of insights that would help shape the future of post-adoption services and supports. Given such data elements as subsidy amounts, their basis, and reasons for changes; duration of subsidy; basis for subsidy amounts at the time of adoption and later; prior foster care payments to the family; linkages to vendor payment files, and reasons for subsidy termination, it would be possible to determine the duration of subsidies; the total amount of a child’s subsidy; and the reasons that subsidies stop, start, or change. In addition, subsidy data should be stored in ways that ensure confidentiality but allow for retrieval for purposes of managing the program, with linkages to the child’s foster care record and detailed information on vendor payment. Information should be stored in a format that maintains historical data to support longitudinal analysis.
Develop programmed child and family assessments. Use of child and family assessments that have been programmed into computers, which can be linked to a computerized case management system, could benefit both case workers and evaluators. Such a system could notify the case manager when a follow-up assessment is needed, allowing assessments during home visits (using a laptop). Data could be transmitted into the database system at the office, where it would be aggregated for evaluation purposes.
Conduct rigorous evaluations. Enhanced information resources should expand the possibilities for a multisite experimental design. Rigorous evaluation would start the long process of determining whether PAS are effectively helping families. Several possible approaches should be considered:
Promote evaluation as a tool for program improvement. Patton’s (1997) utilization-focused evaluation approach stresses the importance of engaging the primary users of evaluation in every step of the process in order to build support for its use. Stakeholders include not only representatives of funding agencies and program coordinators, but front-line staff who implement the program. Focusing the evaluation on the questions they consider critical will improve both its relevance and implementation. Program staff can play a vital role in identifying questions that will complement standard designs and increase the relevance of evaluation to practice.
| Strategic funding and evaluation design can reduce barriers and create incentives for evaluation. |
Structure evaluation processes so that they are useful to programs and families. A related recommendation is to ensure that evaluation processes provide useful feedback to participants. A major barrier to evaluation among PAS program staff was the belief that families were being asked to spend time completing instruments without receiving any direct benefit in return. The choice of instruments should favor those that can provide useful feedback to program staff and families. This will also help mitigate the sense among program staff that evaluations compete with program activities for scarce resources. As noted in the case study, many adoptive families have difficulty obtaining assessment services and interpretation of clinical data. Although evaluation instruments would not substitute for a comprehensive assessment, feedback on the information collected is likely to be perceived as valuable information by many families.
Earmark funds for evaluation. PAS programs need funding that is specifically designated for evaluation and related activities. Without separate evaluation funds, many program leaders will choose to use all, or nearly all, of their resources for services to families and children. Earmarking funds for evaluation will convey the fact that funding agencies (at both the federal and state levels) view evaluation as essential. Designating funds will also help mitigate concerns by program coordinators that evaluation takes resources away from needed services. Program leaders would then be held accountable for allocating those resources for evaluation.
Fund programs for multiple years. Short funding cycles make it difficult to plan, implement, and evaluate programs in the time allotted, so that managers are unlikely to invest in evaluation staff and activities. Funding programs for four years or longer ensures that they have sufficient time to develop, implement, learn from their evaluations, and incorporate those lessons into ongoing practice. Extended funding also provides opportunities for PAS programs to conduct follow-up activities, producing more substantive evaluations and facilitating assessment of outcomes.
Provide evaluation technical assistance. Accessible, culturally appropriate technical assistance can be used to supplement PAS programs’ evaluation skills, or to build long-term evaluation capacity within the organization. Depending on the program’s needs, technical assistance may emphasize support (where the provider conducts some of the evaluation activities with input from the program) or capacity building (where the provider trains and coaches program staff who carry out the evaluation). Technical assistance should be tailored to the particular needs and interests of the program, and may include evaluation design, development or selection of data collection tools, data management and analysis, and application of findings to program development.
Main Page of Report | Contents of Report
Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)