| [Previous] | [Table of Contents] |
The HomeRebuilders project was an ambitious effort to test a major reform of the foster care system in New York City. Planning for the project extended over more than 2 years and involved many staff from the state, city, and the private agencies delivering services. The project entailed countless hours of effort, required negotiations of new relationships among the organizations involved, and took all participants into uncharted territory. It was the first large-scale experiment with managed care concepts in child welfare, and it may still be the largest.
The HomeRebuilders project was built on the notion that the current financing arrangements for foster care, primarily payments for each day in care, create all the wrong incentives for agencies providing the service. Criticisms of per diem payments are based on the view that they do not encourage agencies to work toward reunifying the child with birth parents or toward other resolutions, such as adoption. In fact, the incentive is to retain the child in care because a child in care assures a steady flow of dollars into the program. To correct these perceived distortions in incentives, HomeRebuilders offered agencies capitation payments, a central feature of managed care programs. Agencies were paid a flat amount to work with a group of children who were in foster care at the end of June 1993. Instead of payments for each day in care, the agencies received the predetermined funds, no matter how long the children were in care. Agencies thus assumed the financial risk that children might stay in care longer than projected. The funds were to be used for the costs of foster care, but could also be used for services designed to effect reunification or other permanency arrangements. Funds could also be used for aftercare services, intended to prevent reentry to foster care. The agencies believed that the new arrangements gave them an opportunity to develop new ways to work with families.
As in any major initiative, HomeRebuilders encountered some problems along the way. Agencies did not receive promised funding from Medicaid. During the experiment New York City adopted a new focus on adoption. The shift of a substantial number of cases in the HomeRebuilders group to adoption created financial pressures, since adoptions cost more than returns home (in part, because they take longer on average). There was also a significant decrease in numbers of children entering foster care. Although not directly affecting the HomeRebuilders program, the decrease caused financial stress in the agencies involved and required shifts in personnel and layoffs. Finally, and perhaps most important, the program was terminated after two and a half years, 6 months before its scheduled ending date, causing considerable disruption in the agencies.
It should be noted that the agencies insisted to us that HomeRebuilders was not a managed care program. In the pure form of managed care, agents given responsibility for cases also have the authority to make critical decisions. In HomeRebuilders, agencies had to have the approval of the New York City CWA and the court to return children home, close the case, change the goal to adoption, and to take other actions in the case. Thus, there were risks that were imposed on the agencies that were not under their control, in particular, the possibility that a child would remain in care longer than the agency thought was necessary. The agencies had to bear the financial effects of these longer stays.
As indicated above, the focus of analysis of outcomes was on differences between experimental and comparison groups within agencies. There is considerable variation in results between agencies, such that combining across agencies would not be meaningful. Furthermore, one must distinguish between "true experimental agencies" in which random assignment to experimental and control groups was employed and the other agencies in the project.
Discharge and Days in Care. The data indicate that the New York HomeRebuilders program had the effect of speeding up discharge and reducing days in care in one of the three agencies using random assignment of cases (LF). In that agency, the difference between experimental and comparison groups in days in foster care at the end of the program was on the order of 13 percent. A 13% improvement is not large, but it is larger than the program planners anticipated for the program as a whole (planners had hoped for about a 10% reduction). If this percentage had held generally, one could point to the substantial dollars that such an improvement would represent. Since it did not hold generally, claims of potential large cost savings cannot be supported. In the other two random assignment agencies discharge proportions and days in care did not significantly favor the experimental group. In one of the nonrandom assignment agencies (NYF), discharge rates were higher and days in care lower for the HomeRebuilders group compared with the comparison group. Because the groups were not randomly formed, the possibility that these differences are due to characteristics of the groups cannot be excluded. However, the observed effect survived a regression analysis using available administrative data on a number of control variables, thus providing some reason to believe that the differences in NYF reflect a real effect of the program. JC had high proportions of cases closed and low average days in care, but definitive conclusions cannot be drawn about those results because there was no comparison group in that agency.
Child and Family Functioning. The caretaker and caseworker interview data reveal few effects of the program on family and child functioning. There are always many possible explanations for findings of no difference: the intervention theory was faulty, the theory was inadequately implemented, the program had positive effects but they were not measured, the measures that were used lacked reliability or validity, experimental conditions were not maintained (e.g., the experimental group did not receive the intended intervention or the comparison group received some or all of the services intended for the experimental group), or the experiment lacked statistical power to detect effects (because numbers of cases were small, a particular problem when the expected effects are small). There may also have been major shifts in the environment of the program; shocks to the system that seriously affected the outcome.
It is possible that these and other factors played at least some part in the lack of detected effects on family and child functioning. One aspect of the data collection should be noted: interviews were held with caretakers and caseworkers 6 to 12 months after the end of the program. Furthermore, many cases had closed much earlier and in those cases the time between the last contact with the agency and the interviews was even longer. Thus, the interview data are subject to problems of recall. There were also problems of low sample size. Although the overall number of cases in the experiment was fairly large, subgroups within agency experimental and control groups in the interview data were quite small. In the realm of program implementation, there were a number of problems, noted above.
As suggested earlier, there can be argument about the interpretation of findings of no effect on child and family functioning. On the one hand, one would like to believe that programs such as this have positive effects on functioning. However, it can be argued that lowering care days without adversely affecting functioning is an accomplishment. It is often claimed that managed care programs, because of their focus on fiscal outcomes, fail to maintain quality of services, to the detriment of clients.
Can the data be used to throw light on this proposition? The issue can be explored in LF, where there was a lowering of care days and comparative data on functioning is available from interviews with caretakers and caseworkers in the experimental and control groups. Can it be shown that no harm was done to children and families in the HomeRebuilders groups in this agency? The problem faced here is classical in statistical analysis: it is quite difficult to "prove the null hypothesis." Finding no significant difference in functioning between the experimental and comparison groups is not the same as demonstrating that there was no difference. There are a number of other explanations for lack of significance, as indicated above. In particular, using no significant difference findings as evidence for no real difference requires high statistical power. Power in these comparisons suffers because of low numbers of interviewed caretakers and caseworkers. Furthermore, the nonsignificant differences in means and proportions of functioning variables often favor the comparison group.
A second way to explore the hypothesis of no harm to the experimental group would be to compare the variances of outcome measures in the experimental and comparison groups. It is possible that the new service benefited some families while causing deterioration in others, so that the average outcomes were about the same as in the comparison group. This would be evidenced by a larger variance in the outcome measures in the experimental group. Conversely, it is possible that the program resulted in less variation in outcomes, although the average outcome was unaffected, so that fewer people in the experimental group were at the "low end" (fewer people would be at the "high end" as well). The variances of the family and child functioning outcome variables were examined to test these possibilities. Unfortunately, the results were quite inconclusive. More often than not, the variances in the experimental group were higher than in the comparison group, but the differences were rarely significant. The findings are not conclusive evidence that harm was caused, nor do they show that it did not occur.
Still another refinement would be to examine the functioning of families and children in experimental group cases in which children were returned home. This is the group that would be most likely to experience the hypothesized detrimental effects of managed care since some of these cases might have been returned home too quickly because of the financial incentives of the program and might have suffered as a result. Efforts to explore this possibility also were inconclusive.
Reasons for Lower Care Days. What were the reasons for the higher discharge rate and lower days in care in the LF (and possibly the NYF) experimental groups? The causal factors cannot be specified with precision, but some hypotheses will be presented about what the critical elements might have been, hypotheses that need further testing.
The intervention in HomeRebuilders may be thought of as consisting of two main aspects: an altered financing mechanism and a different approach to practice. The approach to practice may be seen as being motivated by the shift in financing, although many agencies had previously contemplated changes in their approaches. It is clear that the financing mechanism was implemented more or less as intended, except that the project was terminated 6 months before the agreed ending date. The approach to practice called for more intensive work with HomeRebuilders clients, lowered caseloads, quicker decisions to confirm or change initial permanency goals, and involvement of specialists in such areas as housing and substance abuse.
Interviews with caseworkers provide some evidence that the HomeRebuilders group in LF received somewhat more extensive service than the comparison group; results were in the same direction for MM and HD, but were not statistically significant (because comparison group caseworkers were not interviewed in NYF, the group cannot be compared in that agency). Unfortunately, these results were not confirmed in the caretaker interviews. Caretakers in the experimental and comparison groups did not report receiving different levels of service in any agency. It is possible that the failure to find differences in the caretaker interview data is due to problems in recall or other difficulties in data collection mentioned above.
It is thought that one of the most important aspects of reunification work is aftercare services, that is, services to prevent reentry to foster care provided to families following the return of children (and possibly to adoptive families after the adoption). The agencies stressed that the availability of aftercare services enabled HomeRebuilders workers to take greater risks in returning children home. Neither the caseworker nor caretaker interviews uncovered differences between experimental and comparison groups in level of aftercare services. In any event, rates of reentry to foster care were very low in both groups. The contribution of services to that outcome is unknown. Beyond the interviews with clients and caseworkers, there are other data from interviews with program directors on the nature of programs in the agencies. In all of the agencies, it is clear that workers were dedicated to their tasks and invested in helping clients improve their lives. In the LF HomeRebuilders program a number of specialists in housing, nursing, budgeting, education, and psychology were hired. JC hired parent advocates. Specialists were also hired in the other agencies, but they sometimes worked on behalf of both the experimental and comparison cases. In the LF experimental program, there appears to have been extensive worker training, perhaps more than in the other agencies. Experimental group workers in LF and HD were paid a small salary premium for their work on the project. In all of the true experimental group agencies (LF, HD, and MM), experimental group workers had somewhat smaller caseloads than did control group workers.
Although all of these elements might help explain the observed effects on time in care, there is more to the story. Activities with clients may be thought of as composed of services provided and decisions made. In part, the altered financing mechanism was intended to change the ways that decisions were made about cases, in particular, to speed them up when possible. There is evidence that in the LF HomeRebuilders program there was a systematic effort to assess the caseload at the beginning of the project. An important aspect of this assessment was triaging, that is, determining the group of cases most likely to be able to return home quickly and directing initial efforts to that group. This contrasts with the approach often taken in social programs, in which the greatest efforts are expended on those cases that are most difficult -- those most in need. But those most in need are not necessarily those most likely to benefit from a program. These observations reflect a general sense that the LF HomeRebuilders program displayed considerable clarity of purpose.
Other agencies also engaged in efforts to set priorities among cases. JC determined which families needed housing and established a goal of finding homes within a short period of time. MM identified cases with one pending major need (e.g., housing) with the intent of focusing on those cases first; however, it is not clear whether these priorities were consistently implemented.
Some observers have suggested that another possible contributing factor to effectiveness is the availability and use of data systems. LF was reported to have quite a good system for tracking cases and utilized the system in sophisticated ways. The effective management of programs such as this obviously depends on ready accessibility of both individual and aggregate information on cases.
The differences among agencies in the use of adoption as a permanency outcome require some comment. Originally, HomeRebuilders was conceived as a program to reunify children with their birth parents, and the target group was specified as children in family (nonkin) foster care with a goal of return home. That conception was broadened to include adoption as a possible outcome, and some of the agencies accepted cases into the project with goals of adoption. In four of the five agencies with comparison groups, the experimental program used adoption less frequently than the comparison program. Fewer children were adopted in both groups in LF, but adoption was particularly rare in the LF experimental group (32% of cases closed in the control group were adoptions compared with 9% in the experimental group). NYF, on the other hand, had a far higher number of adoptions (70% in the comparison group, 67% in the experimental group). The differences appear to reflect differences in values. In LF, we heard strong opinions that work should be focused on returning children to their birth parents and that adoption was a much less desirable outcome. Evidently the approach in other agencies, particularly NYF, was different. Of course, this suggests that the outcomes that families are likely to experience depend very much on the agencies with which they are assigned to work.
Additional analyses could be done on these data. Data obtained from caretakers on social support have not been analyzed. The data on services might be examined further, for example, determining the relationship between services generally and particular types of service and outcomes. The role of foster parents and of visitation might be more extensively examined.
But the data of this study are limited, particularly because the interviews were retrospective, conducted sometimes long after caretakers and caseworkers were involved in services. Prospective studies of managed care efforts, studies employing strong evaluation designs, preferably with randomized comparison groups, are needed. Data need to be collected from participants (perhaps including children) before, during, and after the intervention.
The results suggest further study of a number of issues. Prominent here is the question of the conditions under which changes in fiscal incentives and other features of managed care have the effects theoreticians ascribe to them. Since social programs are never implemented under ideal circumstances, there is a need to better understand the interaction between new programs and context. Beyond that, further study is needed of important components of reunification programs: how critical decisions get made, the importance of intensive services, the role of foster parents, and the place of aftercare services. Future experiments in this area should explore the use of clear protocols for decision-making about cases, the use of triaging strategies, and the availability of data systems for case and program management.
A primary goal of initiatives like HomeRebuilders is to reduce days in foster care. The principal mechanism by which this is to be achieved is a shift in the fiscal incentives provided to agencies. The findings of this evaluation suggest that the goal is likely to be achieved only under certain circumstances: in agencies with particular characteristics or with particular approaches to the work. Among necessary preconditions, the data suggest the importance of clear decision-making protocols, triaging strategies, and data systems that can be used for case and program management. Hence, changes in fiscal incentives do not appear to be sufficient causes of major reform in child welfare. Although the flexibility in funding provided the opportunity for agencies to develop creative approaches to service delivery, one might also ask whether such funding changes are necessary for reform, or whether other approaches might achieve the desired benefits. There are current in the field proposals for major change that do not place fiscal reform at the center of concern, most notably the shifting of responsibility for response to child maltreatment to communities. At the moment, there is no dominant approach to change in the child welfare system that has been empirically demonstrated to be superior. More experimentation is needed, with both managed care and other approaches.
| [Previous] | [Table of Contents] |