Evaluation of Family Preservation and Reunification Programs: Interim Report. 9.2 Targeting

01/08/2001

The findings of no effects of family preservation programs on placement rates and of problems in targeting these programs are not new, they have been observed in a number of rigorously designed experiments.6 Partially as a result of these previous findings, efforts were made in this project to improve targeting. In New Jersey and Kentucky (but not in Tennessee) a screening instrument developed by the evaluators was employed to encourage referral of cases with a risk of imminent placement and to discourage referral of cases not at risk of placement. It is evident that this effort did not work; evidently, the screening instrument was a weak "intervention" in the problem of targeting.

Clearly, referring agents sent families to the programs that did not fit the criterion of imminent risk of placement. Our interviews with referring workers, discussed in earlier chapters, reveal some of the reasons. Workers acknowledged that they often did not refer cases that were at risk of placement, rather they used the programs for families that they thought could benefit from them. Evidently, they believed that in cases where placement was needed, family preservation services were not appropriate, contrary to the assumptions of the designers of these programs. But the programs were valued, and they were used to help families in the context of a generally service-poor child welfare system.

Table 9-2.
Summary of family and child functioning outcomes, data from caretaker interviews
Differences between experimental and control groups at post treatment, follow up, and change over time
Area Post treatment Follow up Change over time
Life events
Positive life events scale (6 items) No significant differences between experimental and control groups in any state No significant differences between experimental and control groups in any state No significant differences between experimental and control groups in any state
Negative life events scale (8 items) No significant differences No significant differences No significant differences
Depression scale (4 items) No significant differences No significant differences No significant differences
Family problems, 9 individual items KY: no significant differences NJ: fewer experimentals did not have enough money for food, rent, or clothing TN: fewer experimentals had few or no friends No significant differences NA
Economic functioning (4 items)
Individual items KY: no significant differences NJ: experimental group had less difficulty paying rent and buying clothes TN: no significant differences KY: no significant differences NJ: no significant differences TN: fewer experimentals had difficulty paying rent NA
Scale KY: no significant difference NJ: experimental average lower (better) TN: no significant difference No significant differences No significant differences
Household condition (10 items)
Individual items KY: experimentals had fewer broken windows or doors NJ: no significant differences TN: more experimentals in unsafe building because of illegal acts No significant differences NA
Scale No significant differences No significant differences No significant differences
Child care practices (15 items)
Individual items KY: fewer experimentals used punishment for not finishing food NJ: experimentals less often got out of control when punishing child and more often encouraged child to read a book TN: more experimentals went to amusement park, pool, or picnic No significant differences NA
Positive scale (5 items) No significant differences No significant differences No significant differences
Negative scale (10 items) KY: no significant difference NJ: experimentals lower (better) TN: no significant difference No significant differences No significant differences
Punishment (5 items) KY: no significant difference NJ: experimentals lower (better) TN: no significant difference No significant differences No significant differences
Caretaker depression scale (13 items) No significant differences No significant differences No significant differences
Child behavior
Aggression scale (3 items) No significant differences No significant differences No significant differences
School problems (5 items) No significant differences No significant differences No significant differences
Positive child behaviors (10 items) No significant differences No significant differences No significant differences
Negative child behaviors (21 items) KY: no significant differences NJ: experimental group lower (better) TN: no significant differences No significant differences No significant differences
Overall assessment of improvement KY: experimentals, greater improvement NJ: experimentals, greater improvement TN: no significant difference No significant differences NA

Note: Changes over time were not determined for individual items so entries for those cells are designated "NA."


Table 9-3.
Summary of family and child functioning outcomes, data from caseworker interviews
Differences between experimental and control groups at post treatment and change over time
Area Post treatment Change over time
Caseworker report of caretaker functioning (9 items)
Individual items KY: no significant difference NJ: control group higher (better) in ability in giving affection and providing learning opportunities TN: experimental group higher (better) on five items KY: respecting child's opinions: experimental group declined, control group increased NJ: control group had more positive change in respecting child's opinions TN: experimental group more positive change on setting firm and consistent limits
Scale KY: no significant difference NJ: no significant difference TN: experimental group higher (better) No significant differences
Caseworker report of household condition Scale (13 items) KY: control group better NJ: control group better TN: no significant difference No significant differences
Caseworker report of caretaker problems Scale (21 items) KY: experimentals more problems NJ: no significant difference TN: no significant difference KY: no significant difference NJ: no significant difference TN: experimentals declined more
Caseworker report of child problems Scale (12 items) No significant differences No significant differences

There are other possible explanations for the low placement rate in the control group. It is possible that in cases assigned to the control group, workers on those cases exerted efforts to prevent placement of the child. Placement prevention as a central value may pervade the system (perhaps more during the time we were collecting these data than now, it is possible that the Adoption and Safe Families Act has shifted emphasis away from this value). Of course, in this regard the philosophy of family preservation seems to have been widely adopted, even though rigorous evaluations have not shown placement prevention effects of its services.

But there are still other aspects of the targeting problem. Homebuilders has developed into a quite generalist program, used in a wide variety of cases. In Kentucky and New Jersey there is considerable heterogeneity in the cases referred to these services, in both characteristics and problems of families and in where the case is in the child welfare system. Families come from both the investigative and on-going phases of cases.7 It seems likely that many of those referred from on-going caseloads are not referred because of likelihood of placement but because the case is not going well and everything else has been tried.8 Families do not always appear to be in crisis, another important criterion for referral. Furthermore, a number of cases do not involve abuse or neglect, but rather are cases of child dependency or of parent-adolescent conflict. And the cases involve a wide range of ages of children at risk. It could be argued that this variation is detrimental to the development of programs. No one program can expect to be successful in all cases. Having such variation inevitably results in a lack of focus and prevents the development of specialized expertise in handling particular cases. The lack of focus and expertise is likely to affect the outcomes that can be expected. Furthermore, the variation in the character of cases must contribute to variations in outcomes.

A natural response to this state of affairs is that we must tighten up the targeting, demanding strict adherence to referral criteria. Our attempts to assist states to do this were clearly unsuccessful. We suggest that it will be extremely difficult to achieve the goal of better targeting. There are a number of reasons for this skepticism. Referring workers acknowledged that they often referred families that were not at risk of placement, at least not those at imminent risk of placement. We cannot fully explain why workers did not follow the rules for referrals, but we can propose some conjectures. Workers believe that they remove children from the home only when that is absolutely necessary, when no service can prevent placement. In this sense, one might conclude that family preservation values have come to pervade the system, there are few unnecessary placements, leaving few placements to be prevented with intensive services9 However, these services are valued by referring workers, they are responses to the needs of families (families other than those with children about to be placed), and services to meet those needs are scarce. Hence, family preservation programs are used for very real needs of families in the child welfare system.

Beyond this dynamic, there is the general tendency to expand the benefits of a good program. If a program is believed to be beneficial, it is often assumed that it will be useful for an ever-expanding range of cases. Evidently, this occurred in the states we studied. Expansion of the target group is aided by the fact that target group definitions usually have one or more vague terms that allow for the expression of discretion (e.g., most people’s problems can be conceptualized as "crises").

Finally, our efforts to identify particular groups of families for which the programs are successful at preventing placement were mostly unfruitful. Hence we are unable to satisfy the demands of policy makers and practitioners for guidance on specific groups that might be targeted.

These circumstances, together with the fact that referrals to family preservation programs involve judgments that cannot be completely systematized or circumscribed, lead to our skepticism about the likelihood of improving targeting of these programs. Furthermore, it is possible that the programs are, by and large, being used in those circumstances for which they are best suited.10


(6) J. Littell and J. Schuerman. (1995). A Synthesis of Research on Family Preservation and Family Reunification.http://aspe.hhs.gov/hsp/cyp/fplitrev.htm

(7) A number of cases in Kentucky and New Jersey were referred to family preservation to assist in the return home of children from foster care. These reunification cases were excluded from the experiment, but they may have contributed to the diffusion of the program. 

(8) The fact that the case is not going well and that everything else has been tried may or may not mean that placement is likely. Note that "everything else has been tried" is sometimes specified as a criterion for Homebuilders referral. As we noted earlier, this criterion conflicts with the objective of immediate response to crisis.

(9) However, it is clear that there is great variation among jurisdictions, workers, judges, and other decision makers in the circumstances in which children are removed from their homes (J. Schuerman, P. Rossi, and S. Budde. (1999). Decisions on Placement and Family Preservation. Evaluation Review 23:599-618).

(10) We note one effort to solve the targeting problem in the family preservation program in Detroit. As part of an experimental evaluation of the Families First program, judges were asked to identify cases in which they intended to remove a child from the family, but which they deemed could be diverted to family preservation. After screening by project personnel, a group of such cases was randomly assigned to family preservation or to other services, presumably placement. Results of the study have not been published to date. Our understanding is that the group selected for random assignment was a relatively small portion of all families designated for placement. Furthermore, although the procedure was very promising from the standpoint of tightening up the evaluation, it is unlikely that it could be implemented widely or consistently to solve the targeting problem.