[ Main Page of Report | Contents of Report ]
The purpose of this paper is to review what must be learned to develop more effective programs for assuring that children taken into foster care have permanent, nurturing arrangements for their care. It considers programs designed to enhance the likelihood of reunification with original caretakers but goes beyond that to look at programs focused on permanency more broadly. It also broadens the focus beyond the evaluation of particular programs to consider other empirical work that might inform policy making and program design.(1)
Our discussion is framed by the character of the current child welfare system, the values that drive it, the legislation that guides its operation, and the nature of practice within it. In the last decade there has been a shift away from emphasizing efforts to preserve or restore families whenever possible to emphasizing the importance of assuring the safety of children and achieving permanency for them as soon as possible. The shift in emphasis is reflected in and promoted by the 1997 federal Adoption and Safe Families Act (ASFA) which affirmed the primacy of safety in determining the best interests of the child and placed limits on the time spent on attempts to reunite children with original caretakers before implementing other permanency options. However, when most children are removed from their homes, the initial case goals are still return home.(2) Child welfare professionals have hierarchies of permanency options in their minds, and although there is not universal agreement on a single hierarchy, return home is thought to be the best alternative whenever possible. In this sense, a central requirement for the evaluability of programs is at least minimally met: the necessity for agreement among relevant stakeholders as to objectives of programs.
The discussion here is guided by an updated review of the reunification and permanency practices and programs in 25 states originally included in the 1995 Review of Family Preservation and Family Reunification Programs undertaken as part of the Evaluation of Family Preservation and Reunification Programs. The program update was supplemented by site visits to four programs. We also undertook an examination of decision making processes in three agencies in the Washington, DC vicinity. A framework for what we present here is provided in section 2 of this report entitled "Permanency: A Balancing Act."
The legislation of 1997 not only shifted emphasis back toward the safety of the child, it also placed restrictions on the practice of child welfare workers by limiting the time for decisions, requiring concurrent planning, and limiting the exercise of discretion. Legislative requirements of this kind are always blunt instruments for the achievement of policy objectives. No universal prescription can possibly fit all of the familial circumstances of children in foster care. Fully individualized plans for families are no longer possible.
The requirement that proceedings for termination of parental rights begin if the child has been in state custody for 15 of the most recent 22 months has undoubtedly benefited many children who would otherwise have remained in foster care for a long time. But there are cases in which the legislative timetable may not be appropriate (as is acknowledged by the fact that exceptions are permitted with the concurrence of a judge). Time has different meanings at different ages, and the urgency to act quickly may not be as great for older children as for infants and toddlers. Furthermore, termination of parental rights may not be as appropriate for teenagers in placement who may not wish to sever parental ties and who may be content to live out their childhood in foster care, sometimes with a relative. The requirement for termination may also not be appropriate for some younger children in relative foster care where there are comfortable, or at least adequate, relationships among the natural parents, the foster parents, and the child. This is one of the motivations for the development of the status of guardianship.
One can imagine a number of side effects of the new timetables for decisions. We cite two here. As many children are more quickly discharged from care, whether to their parents, relatives, or adoption, that those remaining in care (1) will be those with problems that are more difficult to resolve such as older children,or special needs children; and (2) will have been in care longer. Hence, the lengths of time in care for the group in care at any one time may be expected to increase (this appears to be happening in some places).
In considering the legislation, one must remember that no policy choice can eliminate all risk of harm to children. Decisions entail predictions and prediction is not perfect. As long as any children are returned home, at least a few will be subsequently harmed. And some children in other permanency arrangements, whether adopted, in homes of relatives, or in long term foster care, will also suffer from maltreatment. And, these arrangements may not turn out to be permanent. As more children enter these alternative arrangements, the rate of their disruption can be expected to increase.(3)
Tracking Studies. The issues identified here make clear the need to continue to track the experience of children and families under the new policy. This should be done first through administrative data systems, which in most states are now improving to the point that long-term follow-up of cases will be possible. But administrative data systems are limited in the information they contain, so they must be supplemented by more detailed studies of the course of cases. How many cases are being declared exceptions to the time limits? In how many other cases do the rules not fit? What is the trend in the rate of maltreatment of children returned home? What is the safety record of alternative placements? We know that decisions on permanency are being made more quickly, will that trend be sustained? Do quicker decisions in fact result in real permanency, or is there an increase in the disruption of placements intended to be permanent? Is the trend toward increased adoptions sustained or will the number of adoptions begin to decline once backlogs of children in care for a long time are reduced?
Decision Making Studies. Discussions of the operation of the child welfare system inevitably lead to questions about decision making. The questions are both descriptive and prescriptive: how are child welfare workers, supervisors, administrators, and judges making decisions in the changed policy environment and how should they be making them? Prediction and risk assessment are central here. Most, if not all, child welfare departments make use of risk assessment tools. While there has been a considerable amount of research conducted in connection with these instruments, we need more. The effectiveness of risk assessment procedures is still not known with certainty and undoubtedly can be improved.(4) Any decision as to a course of action implies predictions of the likely outcomes of both the action to be taken as well as alternative actions. We need to understand better how good those predictions are.
Beyond this, we need greater understanding about how workers make decisions about cases. Again, there is a fair body of literature on the subject, but we need to understand how workers are deciding in the changed policy context. For each of the decisions they face, determinations of services to be delivered to families, deciding to return children home, deciding on the acceptability of a relative's home, moving to terminate parental rights, deciding on an adoptive family, we need to know the factors that enter into the decision, how workers combine those factors in their minds, and how the agency, community, and policy context affects decisions.
Implementation Studies. Besides leading to better decisions, decision making studies probe the ways in which policies are working or not working. Laws and regulations governing the actions of officials are rarely implemented in exactly the ways intended. Street-level bureaucrats shape, and sometimes even subvert, the intentions of policy makers in myriad ways. Sometimes this is a matter of resistance to change, the desire to continue to operate in familiar ways, using skills that are well ingrained. Sometimes it is a matter of inadequate training on new requirements and new approaches. Sometimes it is a matter of adapting general rules to local situations. And sometimes subversion is used to gain more flexibility in order to better serve clients. We need studies of how workers are implementing the requirements of new laws and regulations.
Implementation of Concurrent Planning. One of the central features of the new approaches is the concept of concurrent planning. Although the initial case goal may be return home, workers are required to develop alternatives to that goal from the beginning, usually by exploring the possibility of adoption. This requirement is intended to save time later on, in case the original plan does not work out (the threat of an alternative to return home may also put beneficial pressure on parents to change), but it creates conflicts for workers as well. Workers must be working on two, conflicting plans. Sometimes, as in the case of an older adolescent, an alternative plan such as adoption is just not realistic. Our exploration of current reunification practice suggests that the concurrent plan requirement is interpreted by workers as requiring that adoption or guardianship by a relative or foster parent be explored. "Stranger adoption" (i.e., adoption by an unknown adult) is often not included in concurrent planning. Some workers, contrary to policy, do not begin to work seriously on alternatives until close to the time of administrative or judicial review, when it appears the family is not ready for the return of the child. We need greater understanding of how the concurrent planning requirement is being implemented on the ground and the ways in which it may be avoided or subverted.
The Time Dilemma. The time dilemma facing workers is often put in terms of the conflict between the child's sense of time and the parent's sense of time. For children, a year in placement is a long time, a time in which new relationships may be formed and old ones fade. It is also a time of uncertainty as to the future and it may be a time in which stable, loving relationships are not there to further growth. But parental change does not happen over night, it takes time to overcome old habits, to learn new ways of functioning and particularly of ways to relate to children. The dilemma is particularly acute in cases of parental substance abuse, the treatment of which is characterized by ups and downs, sometimes repeatedly. How many relapses should be allowed? How much time should be allowed for parents to overcome their addictions? Although there apparently has been some waning of the drug problem in our central cities, it remains a central problem facing families in the child welfare system. The field would benefit from greater understanding of the course of treatment of addictions and from better capacity to predict the likely outcome of drug and alcohol treatment.
[Go To Contents]
Requirements for Evaluation. Successful and useful evaluations require that a number of conditions be met:
Evaluations should provide solid evidence for decisions about the particular program under review and decisions to implement a similar program elsewhere. This requires that evaluations be as rigorous as possible, ideally experiments in which cases are randomly assigned either to a group receiving the service being tested or to a group receiving a different service or no service at all. In the absence of a randomized experiment, it should be possible to implement a convincing quasi-experiment, involving similar groups receiving contrasting services. Besides giving the best possible information on effects of programs, rigorous evaluations maximize the possibility of other learning, such as what can go wrong in program implementation. In any event, the evaluation should be prospective, clients should be followed from the time they enter the program or a contrasting service, through the program and for a reasonable follow-up period, probably not less than one year. Given the length of time required for some actions, such as termination of parental rights, a follow-up period of longer than that would be useful.
While a credible contemporaneous comparison group is essential for a useful evaluation, the set of outcomes examined might depend on available resources for the research. At a minimum, outcomes should include an accounting of the status of children at various points in time, that is, whether in foster care, at home, adopted or freed for adoption, in independent living arrangements, etc. Ideally, assessments should also be periodically performed of the well being of children and the functioning of the families in which they are living. If children are not living in their original homes but reunification is planned, the functioning of the homes to which they are to be returned should also be examined. Sources of data should include administrative data on living arrangements, reports of maltreatment, administrative and judicial actions altering the child's status, and services provided. Again, ideally, periodic interviews should be undertaken with caretakers, workers involved with cases (both in the public agency and in other organizations that are involved), and the child. Data might also be obtained from others in a position to know about the child's well being, such as teachers. Evaluations should also consider cost issues, ideally conducting a cost-effectiveness or cost-benefit analysis.
We note a particular problem in the evaluation of reunification programs: as indicated above, it is the business of virtually the entire body of foster care practice to work toward reunification in the first place and, if that does not work out, alternative permanency arrangements. Thus, many jurisdictions do not have identifiable programs for reunification and permanency, rather seeing all of their "out-of-home" work directed toward those ends.(7) Furthermore, there is a lack of well known, well articulated models of reunification practice that have been implemented in large scale and no single program model has captured the attention of the field as a whole.
There are some identifiable programs(8) around the country, and it would be well to review here some of the dimensions of difference among them:
In addition to evaluating whole programs, it would be useful to assess a number of program components that are increasingly being used in reunification and permanency programs. The provision of a particular program feature could be varied within an overall program, some families receiving it and others not, with the outcomes of the differing conditions examined to determine whether the feature made a difference.(9) We consider here several program components that we believe could be separately evaluated.
Foster Parent Mentoring. Foster parents have long been viewed as potential resources in working with birth parents and the idea of foster parent mentoring is being tried in a number of jurisdictions and programs. These efforts obviously require special training of foster parents. It is clear that not all good foster parents are suited for such a role and anecdotal reports indicate that foster parents often resist involvement with birth parents. It is possible that there is a role conflict between foster parent responsibility for the care and safety of children and becoming involved with birth parents. Some staff have reported an unintended consequence of foster parent mentoring is a decision by birth parents to relinquish children once they become aware of the contrast between them and the foster parents. To our knowledge, no rigorous evaluations of foster parent mentoring have been undertaken.
Family Group Conferencing. Family group conferencing seeks to bring together the family, relatives, neighbors, community supports (such as clergy), and representatives of organizations working with the family to determine goals and courses of action. Theoretically, a conference should occur at the outset of the case and periodically thereafter to assess progress and adjust plans. Some research on the outcomes of family group conferencing has been conducted in New Zealand and in Europe. The Chapin Hall Center for Children is conducting an evaluation of the Edna McConnell Clark Foundation Community Partnerships Promoting Change program, in which family group conferencing is a prominent feature. However, we know of no rigorous equivalent group designs in the United States.
Termination of Parental Rights Mediation. TPR mediation is being tried in a number of jurisdictions as an attempt to avoid adversarial proceedings and shorten the time before children are available for adoption. In theory, TPR mediation could be examined in a controlled experiment, examining the outcomes of time to termination and families' feelings about the process and its outcome. We have not determined the number of families involved in TPR mediation in particular jurisdictions and one difficulty in evaluation may be that the numbers of cases undergoing mediation in a reasonable period of time are too small to support a controlled experiment.
Concurrent Planning. Concurrent planning presents special problems in constructing rigorous evaluations. For one thing, it is expected by policy to take place in most cases, an expectation that would need to be waived for cases in a control group. Furthermore, there are difficulties in determining just when concurrent planning has taken place: what activities constitute a threshold that could be considered concurrent planning and how can it be determined whether or not these activities have taken place? The determination of outcomes may also be problematic. It is expected that cases with concurrent planning will on average achieve a permanent solution more quickly, but presumably this expectation applies only to those cases in which an alternative to return home is decided upon, so the expectation of shorter times to permanency only applies to a subgroup. Complicating interpretation of times to permanency is the possibility of extensions of timelines and delays in obtaining TPR and finalizing adoptions. While randomized experiments on concurrent planning may not be advisable at present, studies of its implementation and process are quite important to determine whether it is being implemented as intended.
Other Components of Permanency Practice. There are a number of other components of reunification and permanency practice that are potential candidates for evaluation. These include assessment procedures, approaches to visitation, the provision of aftercare services (for both reunification and adoption), and recruitment programs for foster and adoptive parents. However, while these are important elements of practice, we do not know of promising new approaches in these areas that might warrant the mounting of rigorous evaluation.
Therapeutic Technical Packages. A number of models and approaches have been developed for working with troubled families and children. They include multi-systemic therapy, attachment trauma therapy, family resolution therapy, parent-child interaction therapy, parents anonymous, and many others. Some of these approaches focus on work with disturbed children and adolescents. Treating behavioral and psychological disturbances of children obviously may have impact on the likelihood of successful reunification, but we have not focused on these issues in this paper, since the field of child welfare more often focuses on parent and family interventions. Technical packages have varying degrees of research support. Multi-systemic therapy is one approach that has been evaluated fairly rigorously in contexts outside reunification and might well be tried in reunification work.
Section 3 of this report, Permanency and Reunification Trends in 25 States, describes programs in 12 locations. Section 2, Permanency: A Balancing Act, puts these programs in four categories:
There are also specialized programs for substance abuse outside the framework of the courts. An example is Mothers Making a Change (MMAC) in Cobb and Douglas Counties in Georgia, a comprehensive program that includes community outreach, early intervention, family preservation and support services, and substance abuse treatment services. The program involves an inpatient phase of 90 days, followed by 9 months of services in the community during which children should be reunified. After that, clients receive 6 months of aftercare services. At the time we talked with this program there were 21 parents in the residential phase and 30 in the nonresidential phases.
All of the above initiatives deserve evaluation. Of particular urgency is the evaluation of drug courts, both because of the centrality of substance abuse as a problem in child welfare and because drug courts represent an increase in responsibilities and influence of the courts in the child welfare system. An assessment of the effects of strong court sanctions is also needed. We understand that an evaluation of the San Diego program is contemplated and that effort should be closely watched.
Other programs focused on substance abuse should also be evaluated. The Georgia Mothers Making a Change effort could be explored further for possible evaluation. A major issue here is program size. The numbers of cases involved at present may not be sufficient to allow for rigorous evaluation. It is also not known whether there are cases referred that cannot be served because of program capacity limits, thus allowing for random assignment or the construction of a comparison group by other means.
Also much needed are evaluations of community based programs. Some are underway. As mentioned above, Chapin Hall is engaged in an evaluation of the Edna McConnell Clark Foundation's CPPC program. The Casey Foundation's Family to Family programs have research components (for example, the Child Welfare Research Center at the University of California, Berkeley is working with the Los Angeles program). To our knowledge, the Lucas County program is not yet being subjected to rigorous evaluation and could be explored further as an evaluation site.
A major difficulty in evaluating community programs is that they are intended to have community-wide impacts or to involve all cases in a community. (The Lucas County program includes all children in care in the county.) Therefore, forming comparison groups is quite problematic, since one cannot identify contemporaneous cases in the same community that are not affected by the initiative. In this situation, comparisons over time are often suggested (comparing outcomes before the program was implemented) but such comparisons are always subject to alternative explanations of the source of the change (economic conditions are always changing and communities do not otherwise stand still). Comparisons may also be made with other communities in which the initiative is absent but such comparisons are also usually flawed, since one cannot be sure the communities were similar before implementation of the initiative or that their trajectories of change would have been the same without the program (communities are usually chosen for a program because of the likelihood of success). It would be possible to randomly assign communities to interventions but so far foundations do not appear to have taken that option seriously (most community intervention efforts have been supported by foundations).
Programs for children with severe mental health needs and wraparound programs should be evaluated. As we note above, the Boston Family Reunification Network has associated research activities. The possibility of an evaluation of the Santa Clara County program should be explored further. It appears that program may be marginally large enough to support rigorous evaluation, but it is not known whether saturation has been reached.
While it would be desirable to conduct rigorous evaluations of family preservation-like reunification programs, it appears to us that research implementation issues may often be quite daunting. In particular, the programs we are aware of are generally quite small, too small to allow for the accumulation of enough cases within a reasonable period of time.
Given the increased interest in subsidized guardianship as a component of permanency efforts, it is important that evaluative information be developed. Fortunately, a large scale randomized experiment on the Illinois subsidized guardianship program is being conducted by Westat for the State of Illinois. Interim findings from that study showed that the introduction of guardianship as a permanency option increased the number of children placed in permanent arrangements without detracting from the number of children adopted. Questions still to be answered include the continued impact of guardianship on permanency rates and the effects of guardianship on child well being.
[Go To Contents]
The core of all efforts to understand permanency and reunification in the current context is careful monitoring of system performance. Analysis of administrative data is central here, but should be augmented by special studies, as indicated earlier. Of particular importance is follow-ups for reasonable periods of time of children in various permanency arrangements, follow-ups that go beyond data contained in administrative systems.
Beyond the tracking of system performance, we need studies in a number of other areas:
These studies, if conducted in the most rigorous manner possible, will provide the solid evidence necessary to make policy and program decisions that will serve the best interests of children and families.
[Go To Contents]
1. We limit ourselves here to services for families in which children have been removed from caretakers, not considering "front-end" services or services to intact families, although such services can properly be thought of as part of a continuum of permanency activities.
2. ASFA does, for the first time in federal legislation, specify conditions under which "reasonable efforts" to return children home do not have to be made, so that in these cases the initial case plan objective does not have to be reunification.
3. It is to be expected that as more alternative arrangements are used, more of them will experience difficulties. But the rate of difficulty is also likely to go up, since thresholds for deciding to use a particular alternative arrangement will go down. That is, use will be made of relative's homes and adoptive homes that might not have been used previously.
4. Determining the effectiveness of risk assessment instruments in a definitive way is quite difficult. Ideally, one would want a design in which cases with similar levels of risk (or predictions) are randomly assigned to alternative courses of action.
5. Negative findings are useful, but more is learned and better guidance for decisions is available when results are positive. An exception is the situation in which the program has been widely adopted or is widely touted. In this case, it is desirable to undertake a rigorous evaluation in order to verify or disprove accepted wisdom.
6. We cannot specify more precisely the sample sizes required without further exploration of expected size of effects of particular programs.
7. This is in contrast to the situation in regard to family preservation a decade ago. Although the ideals of family preservation had pervaded the system, many jurisdictions had (and still have) identifiable family preservation programs, some of them operating at quite large scale.
8. By an "identifiable program" we mean a set of activities involving an identified group of staff who spend a substantial portion of their time (though not necessarily full time) on the work, referral of specific cases, and specification of goals and of activities designed to achieve them.
9. One difficulty with component evaluation is that all elements of a program may be seen as crucial, the program intending to make use of interactions among them, to produce the desired outcomes.
Main Page of Report | Contents of Report
Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)