Requirements for Evaluation. Successful and useful evaluations require that a number of conditions be met:
- The program should have a well articulated set of objectives and a well specified set of activities to achieve them. There should be a "theory of change" that is logical and believable, such that it is credible that the intended objectives will be met. This is a particularly difficult requirement for programs in child welfare, since the problems dealt with are serious and not easily overcome or managed. A corollary of this requirement is that it should be likely that the evaluation will have positive findings, that is, that the program will prove successful.5
- The program should be generalizable. It should be plausible that it could be implemented in other locations with differing contexts, including differing legal and regulatory frameworks. It should be possible to implement the program elsewhere at a scale such that it will make a difference.
- There should be the possibility of implementing a rigorous evaluation, providing the best possible foundation for policy and practice decisions.
- The program should have some promise of cost-effectiveness. Without this requirement it is unlikely that it would be adopted elsewhere.
- The objectives should be measurable, they should be observable within a reasonable period of time after the completion of the intervention and there should be observable indications of their achievement.
- It is necessary that the program be large enough to provide enough subjects for adequate statistical power to detect important effects. This means that it should be reasonable to expect that a sufficient number of cases will enter the program in a reasonable period of time so that an adequate sample size can be accumulated.6 It should be remembered that referrals often drop when an experiment is undertaken.
- If a randomized experiment is to be implemented, it is desirable that the program not have saturation coverage of the eligible cases, requiring some rationing of services, so that the randomized experiment will not result in fewer cases receiving services than before.
- The organizations involved in the evaluation should have adequate administrative data systems that will provide information on important outcomes. Adequate accounting systems should be available to provide information for cost analyses.
- Program managers and relevant agency executives must be supportive of the evaluation and be willing to exert influence to assure cooperation of staff with evaluation requirements that could be somewhat demanding. Staff sabotage of an evaluation is an ever present possibility, so there should be the capacity to implement mechanisms that will minimize it.
- Front-line workers should get regular feedback about the evaluation and data to support their continued participation.
- It should be possible to assess the extent to which a program is implemented as intended, that is, to determine the degree of program fidelity.
Evaluations should provide solid evidence for decisions about the particular program under review and decisions to implement a similar program elsewhere. This requires that evaluations be as rigorous as possible, ideally experiments in which cases are randomly assigned either to a group receiving the service being tested or to a group receiving a different service or no service at all. In the absence of a randomized experiment, it should be possible to implement a convincing quasi-experiment, involving similar groups receiving contrasting services. Besides giving the best possible information on effects of programs, rigorous evaluations maximize the possibility of other learning, such as what can go wrong in program implementation. In any event, the evaluation should be prospective, clients should be followed from the time they enter the program or a contrasting service, through the program and for a reasonable follow-up period, probably not less than one year. Given the length of time required for some actions, such as termination of parental rights, a follow-up period of longer than that would be useful.
While a credible contemporaneous comparison group is essential for a useful evaluation, the set of outcomes examined might depend on available resources for the research. At a minimum, outcomes should include an accounting of the status of children at various points in time, that is, whether in foster care, at home, adopted or freed for adoption, in independent living arrangements, etc. Ideally, assessments should also be periodically performed of the well being of children and the functioning of the families in which they are living. If children are not living in their original homes but reunification is planned, the functioning of the homes to which they are to be returned should also be examined. Sources of data should include administrative data on living arrangements, reports of maltreatment, administrative and judicial actions altering the child's status, and services provided. Again, ideally, periodic interviews should be undertaken with caretakers, workers involved with cases (both in the public agency and in other organizations that are involved), and the child. Data might also be obtained from others in a position to know about the child's well being, such as teachers. Evaluations should also consider cost issues, ideally conducting a cost-effectiveness or cost-benefit analysis.
We note a particular problem in the evaluation of reunification programs: as indicated above, it is the business of virtually the entire body of foster care practice to work toward reunification in the first place and, if that does not work out, alternative permanency arrangements. Thus, many jurisdictions do not have identifiable programs for reunification and permanency, rather seeing all of their "out-of-home" work directed toward those ends.7 Furthermore, there is a lack of well known, well articulated models of reunification practice that have been implemented in large scale and no single program model has captured the attention of the field as a whole.
5. Negative findings are useful, but more is learned and better guidance for decisions is available when results are positive. An exception is the situation in which the program has been widely adopted or is widely touted. In this case, it is desirable to undertake a rigorous evaluation in order to verify or disprove accepted wisdom.
6. We cannot specify more precisely the sample sizes required without further exploration of expected size of effects of particular programs.
7. This is in contrast to the situation in regard to family preservation a decade ago. Although the ideals of family preservation had pervaded the system, many jurisdictions had (and still have) identifiable family preservation programs, some of them operating at quite large scale.
2.1 Program Dimensions
There are some identifiable programs8 around the country, and it would be well to review here some of the dimensions of difference among them:
- Intensiveness. Reunification programs usually involve some degree of intensiveness of service with families, having relatively small worker caseloads. Sometimes they are very intensive, with extensive worker involvement and employing approaches such as wrap-around services.
- Program theories. Programs have differing theories as to the central problems clients face and as to the activities necessary to deal with those problems.
- Use of community resources. Programs can rarely do it all themselves so they make varying uses of community resources. They vary in the extent of dependence on and involvement with other agencies and community organizations.
- Time limits. Programs usually have time limits, although there often are exceptions made to them. Programs based on family preservation models are generally short (e.g., Missouri's Family Reunion, 8-12 weeks; New Jersey's Natural Parent Support Program, 3-6 months). Most other programs last for a year or two (e.g., Florida's Homeward Bound, 18 months; Georgia's Mothers Making a Change, 12-15 months extendable to 2 years).
- Target groups. Programs receive specific referrals for service, but there is great variation in their target groups. One dimension of targeting is the timing of the referral in the course of the case. Programs could take cases at the time of or shortly after placement of a child in foster care, but this appears to be rarely done in specific reunification programs. More often, programs accept cases only after a child has been in care for a time, while others take long-term, difficult to work with cases. Some programs specialize in cases with particular problems, for example, substance abusing parents or mentally ill children.
- Work with foster parents. Some programs make considerable use of foster parents in working with birth parents while others do not. Experience seems to indicate that foster parents must be specially selected for such work. Many foster parents resist involvement with birth parents and are less than enthusiastic about the prospect of the child returning home.
- Emphasis on pre- or post-reunification, or both. Some programs focus on work to prepare families for return of children and do little with families after the reunification. Others do most of their work shortly before and after reunification has occurred (they could be considered post-reunification family preservation programs). Still others work with families both before and after reunification.
- Use of concurrent planning. As described previously, some programs are quite committed to concurrent planning from the beginning of their work with families while others begin it in earnest only when it becomes evident an alternative plan may be necessary.
- Auspices. Some programs are located in public child welfare agencies while others are conducted by private agencies under contract with the public agency.
- Use of various program components. Examples are family group conferencing and visitation. Some programs use family group conferencing while others do not. While visitation is a part of virtually all reunification practice, the emphasis placed on it varies.
8. By an "identifiable program" we mean a set of activities involving an identified group of staff who spend a substantial portion of their time (though not necessarily full time) on the work, referral of specific cases, and specification of goals and of activities designed to achieve them.
2.2 Evaluation of Particular Program Components.
In addition to evaluating whole programs, it would be useful to assess a number of program components that are increasingly being used in reunification and permanency programs. The provision of a particular program feature could be varied within an overall program, some families receiving it and others not, with the outcomes of the differing conditions examined to determine whether the feature made a difference.9 We consider here several program components that we believe could be separately evaluated.
Foster Parent Mentoring. Foster parents have long been viewed as potential resources in working with birth parents and the idea of foster parent mentoring is being tried in a number of jurisdictions and programs. These efforts obviously require special training of foster parents. It is clear that not all good foster parents are suited for such a role and anecdotal reports indicate that foster parents often resist involvement with birth parents. It is possible that there is a role conflict between foster parent responsibility for the care and safety of children and becoming involved with birth parents. Some staff have reported an unintended consequence of foster parent mentoring is a decision by birth parents to relinquish children once they become aware of the contrast between them and the foster parents. To our knowledge, no rigorous evaluations of foster parent mentoring have been undertaken.
Family Group Conferencing. Family group conferencing seeks to bring together the family, relatives, neighbors, community supports (such as clergy), and representatives of organizations working with the family to determine goals and courses of action. Theoretically, a conference should occur at the outset of the case and periodically thereafter to assess progress and adjust plans. Some research on the outcomes of family group conferencing has been conducted in New Zealand and in Europe. The Chapin Hall Center for Children is conducting an evaluation of the Edna McConnell Clark Foundation Community Partnerships Promoting Change program, in which family group conferencing is a prominent feature. However, we know of no rigorous equivalent group designs in the United States.
Termination of Parental Rights Mediation. TPR mediation is being tried in a number of jurisdictions as an attempt to avoid adversarial proceedings and shorten the time before children are available for adoption. In theory, TPR mediation could be examined in a controlled experiment, examining the outcomes of time to termination and families' feelings about the process and its outcome. We have not determined the number of families involved in TPR mediation in particular jurisdictions and one difficulty in evaluation may be that the numbers of cases undergoing mediation in a reasonable period of time are too small to support a controlled experiment.
Concurrent Planning. Concurrent planning presents special problems in constructing rigorous evaluations. For one thing, it is expected by policy to take place in most cases, an expectation that would need to be waived for cases in a control group. Furthermore, there are difficulties in determining just when concurrent planning has taken place: what activities constitute a threshold that could be considered concurrent planning and how can it be determined whether or not these activities have taken place? The determination of outcomes may also be problematic. It is expected that cases with concurrent planning will on average achieve a permanent solution more quickly, but presumably this expectation applies only to those cases in which an alternative to return home is decided upon, so the expectation of shorter times to permanency only applies to a subgroup. Complicating interpretation of times to permanency is the possibility of extensions of timelines and delays in obtaining TPR and finalizing adoptions. While randomized experiments on concurrent planning may not be advisable at present, studies of its implementation and process are quite important to determine whether it is being implemented as intended.
Other Components of Permanency Practice. There are a number of other components of reunification and permanency practice that are potential candidates for evaluation. These include assessment procedures, approaches to visitation, the provision of aftercare services (for both reunification and adoption), and recruitment programs for foster and adoptive parents. However, while these are important elements of practice, we do not know of promising new approaches in these areas that might warrant the mounting of rigorous evaluation.
Therapeutic Technical Packages. A number of models and approaches have been developed for working with troubled families and children. They include multi-systemic therapy, attachment trauma therapy, family resolution therapy, parent-child interaction therapy, parents anonymous, and many others. Some of these approaches focus on work with disturbed children and adolescents. Treating behavioral and psychological disturbances of children obviously may have impact on the likelihood of successful reunification, but we have not focused on these issues in this paper, since the field of child welfare more often focuses on parent and family interventions. Technical packages have varying degrees of research support. Multi-systemic therapy is one approach that has been evaluated fairly rigorously in contexts outside reunification and might well be tried in reunification work.
9. One difficulty with component evaluation is that all elements of a program may be seen as crucial, the program intending to make use of interactions among them, to produce the desired outcomes.
2.3 Evaluation of Programs With Specific Foci
Section 3 of this report, Permanency and Reunification Trends in 25 States, describes programs in 12 locations. Section 2, Permanency: A Balancing Act, puts these programs in four categories:
- Programs for substance abusing parents. Drug courts are a subtype of programs focused on parental substance abuse. An example of a drug court is the Drug Court Recovery Project in San Diego County, California. Drug treatment is provided by several inpatient and outpatient treatment substance abuse centers. Services other than substance abuse treatment are provided through a county case manager. Parents' failure to comply with treatment may result in the court imposing sanctions such as short stays in jail and fines for contempt of court. Continued violations may result in dismissal from the program and evaluation for termination of parental rights. Phase 1 of the program requires a court appearance once a week for 90 days; phase 2 requires a court appearance once every two weeks for 90 days; and phase 3 requires a court appearance once a month for 90 days. An additional feature of the program is the use of "safe houses," transitional housing units for parents and children while parents are undergoing treatment. This is a large program, about 880 cases were being served in July 2000.
There are also specialized programs for substance abuse outside the framework of the courts. An example is Mothers Making a Change(MMAC) in Cobb and Douglas Counties in Georgia, a comprehensive program that includes community outreach, early intervention, family preservation and support services, and substance abuse treatment services. The program involves an inpatient phase of 90 days, followed by 9 months of services in the community during which children should be reunified. After that, clients receive 6 months of aftercare services. At the time we talked with this program there were 21 parents in the residential phase and 30 in the nonresidential phases.
- Programs for children with severe problems. These programs are focused on children with severe mental health problems and are sometimes characterized by "wraparound" services, an effort to bring to bear a comprehensive set of community services to meet child and family needs. Examples are the Santa Clara County, California wraparound services and Boston's Family Reunification Network. Santa Clara County's program accepts children who are in high-end residential care. It is conducted by two private agencies and has the capacity to serve 195 children. Boston's program is conducted by a network of private agencies and tries to reunify children who are in specialized foster care or residential treatment with their parents. The Heller School at Brandeis University conducts research on the program.
- Community-based programs. Community based approaches to child welfare are being tried in a number of communities. The Family to Family initiative of the Annie E. Casey Foundation is a model that is being used in a number of locations. The Lucas County, Ohio Community Development Department uses the Family to Family model to help facilitate reunification. Six neighborhood community centers provide space for visitation, parenting classes, and family conferences. These multi-service agencies also provide recreation, family counseling, health services, and after-school programs. A Community Advocate program provides intensive in-home services when the child is ready to return home. The county has about 500 children in foster care and 500 in relative care. There are between 80 and 100 families in the Community Advocates program.
- Family preservation model intensive services. These programs use family preservation like services, often adapting the Homebuilders model. Caseloads are small (2-5 families), services are intensive and short term. An example is the New Jersey Natural Parent Support Program in which families receive services for 6 months. The program is fairly small, each of six counties serve 9 families at a time.
All of the above initiatives deserve evaluation. Of particular urgency is the evaluation of drug courts, both because of the centrality of substance abuse as a problem in child welfare and because drug courts represent an increase in responsibilities and influence of the courts in the child welfare system. An assessment of the effects of strong court sanctions is also needed. We understand that an evaluation of the San Diego program is contemplated and that effort should be closely watched.
Other programs focused on substance abuse should also be evaluated. The Georgia Mothers Making a Change effort could be explored further for possible evaluation. A major issue here is program size. The numbers of cases involved at present may not be sufficient to allow for rigorous evaluation. It is also not known whether there are cases referred that cannot be served because of program capacity limits, thus allowing for random assignment or the construction of a comparison group by other means.
Also much needed are evaluations of community based programs. Some are underway. As mentioned above, Chapin Hall is engaged in an evaluation of the Edna McConnell Clark Foundation's CPPC program. The Casey Foundation's Family to Family programs have research components (for example, the Child Welfare Research Center at the University of California, Berkeley is working with the Los Angeles program). To our knowledge, the Lucas County program is not yet being subjected to rigorous evaluation and could be explored further as an evaluation site.
A major difficulty in evaluating community programs is that they are intended to have community-wide impacts or to involve all cases in a community. (The Lucas County program includes all children in care in the county.) Therefore, forming comparison groups is quite problematic, since one cannot identify contemporaneous cases in the same community that are not affected by the initiative. In this situation, comparisons over time are often suggested (comparing outcomes before the program was implemented) but such comparisons are always subject to alternative explanations of the source of the change (economic conditions are always changing and communities do not otherwise stand still). Comparisons may also be made with other communities in which the initiative is absent but such comparisons are also usually flawed, since one cannot be sure the communities were similar before implementation of the initiative or that their trajectories of change would have been the same without the program (communities are usually chosen for a program because of the likelihood of success). It would be possible to randomly assign communities to interventions but so far foundations do not appear to have taken that option seriously (most community intervention efforts have been supported by foundations).
Programs for children with severe mental health needs and wraparound programs should be evaluated. As we note above, the Boston Family Reunification Network has associated research activities. The possibility of an evaluation of the Santa Clara County program should be explored further. It appears that program may be marginally large enough to support rigorous evaluation, but it is not known whether saturation has been reached.
While it would be desirable to conduct rigorous evaluations of family preservation-like reunification programs, it appears to us that research implementation issues may often be quite daunting. In particular, the programs we are aware of are generally quite small, too small to allow for the accumulation of enough cases within a reasonable period of time.
Given the increased interest in subsidized guardianship as a component of permanency efforts, it is important that evaluative information be developed. Fortunately, a large scale randomized experiment on the Illinois subsidized guardianship program is being conducted by Westat for the State of Illinois. Interim findings from that study showed that the introduction of guardianship as a permanency option increased the number of children placed in permanent arrangements without detracting from the number of children adopted. Questions still to be answered include the continued impact of guardianship on permanency rates and the effects of guardianship on child well being.