Evaluation of the New York City Home Rebuilders Demonstration. 3.4.1 Outcomes

09/14/1998

As indicated above, the focus of analysis of outcomes was on differences between experimental and comparison groups within agencies. There is considerable variation in results between agencies, such that combining across agencies would not be meaningful. Furthermore, one must distinguish between "true experimental agencies" in which random assignment to experimental and control groups was employed and the other agencies in the project.

Discharge and Days in Care.  The data indicate that the New York HomeRebuilders program had the effect of speeding up discharge and reducing days in care in one of the three agencies using random assignment of cases (LF). In that agency, the difference between experimental and comparison groups in days in foster care at the end of the program was on the order of 13 percent. A 13% improvement is not large, but it is larger than the program planners anticipated for the program as a whole (planners had hoped for about a 10% reduction). If this percentage had held generally, one could point to the substantial dollars that such an improvement would represent. Since it did not hold generally, claims of potential large cost savings cannot be supported. In the other two random assignment agencies discharge proportions and days in care did not significantly favor the experimental group. In one of the nonrandom assignment agencies (NYF), discharge rates were higher and days in care lower for the HomeRebuilders group compared with the comparison group. Because the groups were not randomly formed, the possibility that these differences are due to characteristics of the groups cannot be excluded. However, the observed effect survived a regression analysis using available administrative data on a number of control variables, thus providing some reason to believe that the differences in NYF reflect a real effect of the program. JC had high proportions of cases closed and low average days in care, but definitive conclusions cannot be drawn about those results because there was no comparison group in that agency.

Child and Family Functioning.  The caretaker and caseworker interview data reveal few effects of the program on family and child functioning. There are always many possible explanations for findings of no difference: the intervention theory was faulty, the theory was inadequately implemented, the program had positive effects but they were not measured, the measures that were used lacked reliability or validity, experimental conditions were not maintained (e.g., the experimental group did not receive the intended intervention or the comparison group received some or all of the services intended for the experimental group), or the experiment lacked statistical power to detect effects (because numbers of cases were small, a particular problem when the expected effects are small). There may also have been major shifts in the environment of the program; shocks to the system that seriously affected the outcome.

It is possible that these and other factors played at least some part in the lack of detected effects on family and child functioning. One aspect of the data collection should be noted: interviews were held with caretakers and caseworkers 6 to 12 months after the end of the program. Furthermore, many cases had closed much earlier and in those cases the time between the last contact with the agency and the interviews was even longer. Thus, the interview data are subject to problems of recall. There were also problems of low sample size. Although the overall number of cases in the experiment was fairly large, subgroups within agency experimental and control groups in the interview data were quite small. In the realm of program implementation, there were a number of problems, noted above.

As suggested earlier, there can be argument about the interpretation of findings of no effect on child and family functioning. On the one hand, one would like to believe that programs such as this have positive effects on functioning. However, it can be argued that lowering care days without adversely affecting functioning is an accomplishment. It is often claimed that managed care programs, because of their focus on fiscal outcomes, fail to maintain quality of services, to the detriment of clients.

Can the data be used to throw light on this proposition? The issue can be explored in LF, where there was a lowering of care days and comparative data on functioning is available from interviews with caretakers and caseworkers in the experimental and control groups. Can it be shown that no harm was done to children and families in the HomeRebuilders groups in this agency? The problem faced here is classical in statistical analysis: it is quite difficult to "prove the null hypothesis." Finding no significant difference in functioning between the experimental and comparison groups is not the same as demonstrating that there was no difference. There are a number of other explanations for lack of significance, as indicated above. In particular, using no significant difference findings as evidence for no real difference requires high statistical power. Power in these comparisons suffers because of low numbers of interviewed caretakers and caseworkers. Furthermore, the nonsignificant differences in means and proportions of functioning variables often favor the comparison group.

A second way to explore the hypothesis of no harm to the experimental group would be to compare the variances of outcome measures in the experimental and comparison groups. It is possible that the new service benefited some families while causing deterioration in others, so that the average outcomes were about the same as in the comparison group. This would be evidenced by a larger variance in the outcome measures in the experimental group. Conversely, it is possible that the program resulted in less variation in outcomes, although the average outcome was unaffected, so that fewer people in the experimental group were at the "low end" (fewer people would be at the "high end" as well). The variances of the family and child functioning outcome variables were examined to test these possibilities. Unfortunately, the results were quite inconclusive. More often than not, the variances in the experimental group were higher than in the comparison group, but the differences were rarely significant. The findings are not conclusive evidence that harm was caused, nor do they show that it did not occur.

Still another refinement would be to examine the functioning of families and children in experimental group cases in which children were returned home. This is the group that would be most likely to experience the hypothesized detrimental effects of managed care since some of these cases might have been returned home too quickly because of the financial incentives of the program and might have suffered as a result. Efforts to explore this possibility also were inconclusive.

Reasons for Lower Care Days.  What were the reasons for the higher discharge rate and lower days in care in the LF (and possibly the NYF) experimental groups? The causal factors cannot be specified with precision, but some hypotheses will be presented about what the critical elements might have been, hypotheses that need further testing.

The intervention in HomeRebuilders may be thought of as consisting of two main aspects: an altered financing mechanism and a different approach to practice. The approach to practice may be seen as being motivated by the shift in financing, although many agencies had previously contemplated changes in their approaches. It is clear that the financing mechanism was implemented more or less as intended, except that the project was terminated 6 months before the agreed ending date. The approach to practice called for more intensive work with HomeRebuilders clients, lowered caseloads, quicker decisions to confirm or change initial permanency goals, and involvement of specialists in such areas as housing and substance abuse.

Interviews with caseworkers provide some evidence that the HomeRebuilders group in LF received somewhat more extensive service than the comparison group; results were in the same direction for MM and HD, but were not statistically significant (because comparison group caseworkers were not interviewed in NYF, the group cannot be compared in that agency). Unfortunately, these results were not confirmed in the caretaker interviews. Caretakers in the experimental and comparison groups did not report receiving different levels of service in any agency. It is possible that the failure to find differences in the caretaker interview data is due to problems in recall or other difficulties in data collection mentioned above.

It is thought that one of the most important aspects of reunification work is aftercare services, that is, services to prevent reentry to foster care provided to families following the return of children (and possibly to adoptive families after the adoption). The agencies stressed that the availability of aftercare services enabled HomeRebuilders workers to take greater risks in returning children home. Neither the caseworker nor caretaker interviews uncovered differences between experimental and comparison groups in level of aftercare services. In any event, rates of reentry to foster care were very low in both groups. The contribution of services to that outcome is unknown. Beyond the interviews with clients and caseworkers, there are other data from interviews with program directors on the nature of programs in the agencies. In all of the agencies, it is clear that workers were dedicated to their tasks and invested in helping clients improve their lives. In the LF HomeRebuilders program a number of specialists in housing, nursing, budgeting, education, and psychology were hired. JC hired parent advocates. Specialists were also hired in the other agencies, but they sometimes worked on behalf of both the experimental and comparison cases. In the LF experimental program, there appears to have been extensive worker training, perhaps more than in the other agencies. Experimental group workers in LF and HD were paid a small salary premium for their work on the project. In all of the true experimental group agencies (LF, HD, and MM), experimental group workers had somewhat smaller caseloads than did control group workers.

Although all of these elements might help explain the observed effects on time in care, there is more to the story. Activities with clients may be thought of as composed of services provided and decisions made. In part, the altered financing mechanism was intended to change the ways that decisions were made about cases, in particular, to speed them up when possible. There is evidence that in the LF HomeRebuilders program there was a systematic effort to assess the caseload at the beginning of the project. An important aspect of this assessment was triaging, that is, determining the group of cases most likely to be able to return home quickly and directing initial efforts to that group. This contrasts with the approach often taken in social programs, in which the greatest efforts are expended on those cases that are most difficult -- those most in need. But those most in need are not necessarily those most likely to benefit from a program. These observations reflect a general sense that the LF HomeRebuilders program displayed considerable clarity of purpose.

Other agencies also engaged in efforts to set priorities among cases. JC determined which families needed housing and established a goal of finding homes within a short period of time. MM identified cases with one pending major need (e.g., housing) with the intent of focusing on those cases first; however, it is not clear whether these priorities were consistently implemented.

Some observers have suggested that another possible contributing factor to effectiveness is the availability and use of data systems. LF was reported to have quite a good system for tracking cases and utilized the system in sophisticated ways. The effective management of programs such as this obviously depends on ready accessibility of both individual and aggregate information on cases.

The differences among agencies in the use of adoption as a permanency outcome require some comment. Originally, HomeRebuilders was conceived as a program to reunify children with their birth parents, and the target group was specified as children in family (nonkin) foster care with a goal of return home. That conception was broadened to include adoption as a possible outcome, and some of the agencies accepted cases into the project with goals of adoption. In four of the five agencies with comparison groups, the experimental program used adoption less frequently than the comparison program. Fewer children were adopted in both groups in LF, but adoption was particularly rare in the LF experimental group (32% of cases closed in the control group were adoptions compared with 9% in the experimental group). NYF, on the other hand, had a far higher number of adoptions (70% in the comparison group, 67% in the experimental group). The differences appear to reflect differences in values. In LF, we heard strong opinions that work should be focused on returning children to their birth parents and that adoption was a much less desirable outcome. Evidently the approach in other agencies, particularly NYF, was different. Of course, this suggests that the outcomes that families are likely to experience depend very much on the agencies with which they are assigned to work.

View full report

Preview
Download

"appa.pdf" (pdf, 120.49Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"appb.pdf" (pdf, 623.25Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"appc.pdf" (pdf, 3.4Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®