Evaluation of Family Preservation and Reunification Programs: Interim Report. Chapter 8: The Outcomes

01/08/2001

The outcomes we examined were the placement of children in substitute care following random assignment to the experimental or control group, subsequent reports of maltreatment and a number of measures of child and family functioning. The focus is on comparisons between the experimental and control groups. Analyses we have designated as "primary" were conducted on all randomly assigned cases except those that were determined to be inappropriate referrals. This includes cases in which the assignment was violated (cases assigned to the control group that were given family preservation services) and cases assigned to family preservation that received no or little such service. Insofar as family preservation services have effects, inclusion of these cases in the analysis will tend to reduce the observed differences between the groups. However, the most rigorous approach to analysis requires that we retain these cases in the group to which they were assigned in order to maintain the statistical equivalence of the groups at the outset of the experiment, which is the reason for random assignment in the first place.

It is likely that violations and minimal service cases differ in systematic ways from other cases (perhaps not detected in the measurements of the study), hence, switching them to the other group would result in groups that were not equivalent at the beginning. It can be argued that inclusion of minimal service cases in the experimental group is quite proper on other grounds: the implementation of any program will involve some cases that do not receive the service, and estimates of impact ought to take that into account. We did conduct analyses ("secondary" analyses) in which the violations and minimal service cases were dropped, so as to examine differences between cases that actually received the intended treatment (family preservation or regular services). This analysis must be viewed as only suggestive, since it does not preserve the initial statistical equivalence of the groups created by random assignment. In fact, the results of the secondary analyses were usually similar to those of the primary analysis. We note in footnotes when the secondary analysis differed substantially from the primary analysis. The secondary analysis tables are presented in Appendix I.

Some analyses were also conducted on a more "refined" sample in which we attempted to focus on cases that approached a conception of "ideal" family preservation cases. Family preservation services are designed for families in crisis, presumably the crisis surrounding a recent allegation of maltreatment, the investigation of that allegation, and the threat of removal of a child. Theoretically, this state of crisis makes families more willing to seek and respond to help. As indicated in Chapter 6, many of the families did not appear to conform to this specification of the target group. In Kentucky and New Jersey we looked at two subgroups of cases, those with a recent substantiated allegation recorded in the administrative data (within three months prior to referral to family preservation services) and those in which an investigative worker was involved. In Tennessee, nearly all of the cases came from investigating workers, so we looked at those cases with an allegation within 30 days prior to referral.