Evaluation of Family Preservation and Reunification Programs: Interim Report. Findings

01/08/2001

This evaluation of family preservation programs was designed to assess the extent to which key goals of the programs are being met: the goals of reducing foster care placement, maintaining the safety of children, and improving family functioning.

The assessment of effects on placement and safety of children was based on administrative data which were available on families for at least one year after the beginning of service. Family functioning was assessed through interviews with caretakers at the beginning of service, one month later (at the end of service for the family preservation group), and a year after the beginning of service. Interviews with caseworkers were also conducted at the beginning and one month points.

No significant differences were found between the experimental and control groups on family level rates of placement, case closings, or subsequent maltreatment. There were a few child and family functioning items in which the experimental group displayed better outcomes than the control group in at least one of the states. However, these results did not occur in more than one state. It was found that family preservation programs in two states resulted in higher assessments by clients of the extent to which goals have been accomplished and of overall improvement in their families' lives.

Reducing foster care placement. In none of the three states were there statistically significant differences between the experimental and control groups on family level rates of placement or case closings (see Table 5). In Kentucky, placement rates at the end of one year were 23 and 24 percent for the experimental and control groups, respectively. In New Jersey and Tennessee, the percents were about 28 and 22 percent.

As to be expected with any program, some of the families assigned to family preservation programs did not receive the services or received a minimal dosage of the services. Also, a small number of the families in the control group were actually provided family preservation services. To address these issues, analyses were conducted in which these cases were dropped (secondary analysis). Results of the secondary analyses were quite similar to the primary analyses, also showing no significant differences between the groups in rates of placement.5

The ideal family preservation case is one in which there has been a recent significant crisis in the family, resulting in the maltreatment that triggers the possibility of removal of the child from the home. Subsamples of cases that approached this ideal were examined. Again in these analyses there were no statistically significant differences between the experimental and control groups in placement rates over time.

In addition to placement rates at various points in time, placement was examined in terms of proportion of time in substitute care subsequent to random assignment. No significant differences were found in care days for the families in any of the three states. In Kentucky both the experimental and control group children spent an average of 6 percent of the days subsequent to random assignment in care. In New Jersey, experimental group children spent an average of 6 percent of that time in placement compared to 4 percent for the control group children. In Tennessee, experimental group children spent an average of 10 percent of that time in placement, compared to 5 percent for the control group children.

Table 5.
Summary of Placement Data, Survival Analyses 
Percents of families experiencing placement of at least one child within specified periods of time
Kentucky One month
(%)
6 months
(%)
One year
(%)
  E C E C E C
Primary 6 5 18 18 23 24
Secondary 4 4 13 17 20 24
Refined Analyses:
  • Investigative cases, primary
8 5 16 14 26 15
  • Recent substantiated, primary
6 3 17 12 29 16
  • Petition cases, primary
7 10 14 26 18 33
Including relative placement, primary 8 9 21 25 27 32
Including relative placement, secondary 5 9 14 25 22 32
 
New Jersey One month
(%)
6 months
(%)
One year
(%)
  E C E C E C
Primary 4 6 19 16 28 22
Secondary 3 6 18 16 26 22
Refined Analyses:
  • Investigative cases, primary
3 5 18 13 25 16
  • Recent substantiated, primary
8 5 20 12 27 15
 
Tennessee One month
(%)
6 months
(%)
One year
(%)
  E C E C E C
CORS, primary 11 11 22 19 23 19
CORS, secondary 7 12 18 19 19 19
Including relative placement, primary 11 11 26 21 28 23
Including relative placement, secondary 7 12 20 19 23 21
Refined Analyses:
  • Recent investigation, CORS
7 12 15 15 17 15
  • Recent investigation, includes relative placement
7 12 18 18 22 21

NOTE:  Primary analyses included all cases randomly assigned, except for cases that were determined to be inappropriate referrals. Secondary analyses dropped two categories of cases:  families in the control group that were actually provided family preservation services ("violations") and families in the experimental group that received no or little service ("minimal service cases").  "Refined" analyses were limited to subgroups that were thought to represent better targeted cases.  Most of the analyses above are of records of placements in administrative data.  In Kentucky and Tennessee data were also available from case records on placements with relatives that were not recorded in the administrative data.  Those data are included in the rows labeled "including relative placement."


Targeting. Since these programs were intended to prevent the placement of children, the target group for the services was families in which at least one child was "in imminent risk of placement." As in previous studies, it was found that most of the families served were not in that target group. This is shown by the placement rate within a short period of time in the control group, indicating the placement experience in the absence of family preservation services. In all three states, the placement rate in the control group within one month was quite low. It would, therefore, have been virtually impossible for the programs to be effective in preventing imminent placement, since very few families would have experienced placement within a month without family preservation services.

A number of subgroups that were thought to represent better targeting were examined. These included cases coming directly from the investigation of an allegation of abuse or neglect, cases with recent substantiated allegations of abuse or neglect, and, in Kentucky, a subgroup of cases in which workers had submitted petitions to the court for placement or some other court-ordered intervention. In none of these subgroups did placement rates in the control group within one month exceed 12 percent. Hence, even in these more refined (from the standpoint of targeting) subgroups, the intended target group was not in evidence.

It should be noted that the results found here occurred despite efforts in this project to improve targeting. In Kentucky and New Jersey a special screening form, developed by the evaluation team, was employed to rate the risk to children with the intent that cases with intermediate risk would be referred to the program. In Kentucky efforts were made to divert to family preservation cases that had been referred to the court. In Tennessee, special training efforts were instituted to address concerns about targeting.

Child Safety. Maltreatment subsequent to the beginning of service was generally not related to experimental group membership, except for one subgroup in Tennessee. Subsequent maltreatment was measured by the occurrence or nonoccurrence of a substantiated allegation of abuse or neglect following an investigation of such an allegation. The rate of subsequent maltreatment was relatively low, about 18 percent of the families in Kentucky had a substantiated allegation within one year of random assignment; in New Jersey the rate was 12 percent and in Tennessee, 25 percent. In Tennessee, in those families with an allegation within 30 days prior to random assignment, the experimental group children experienced fewer substantiated allegations than children in the control group did.

The findings of little difference between the experimental and control groups in subsequent maltreatment can be read in two ways. It indicates that families served by family preservation were no more likely than families not receiving the services to be subjects of allegations of harm. In this sense, children were, by and large, kept safely at home while receiving family preservation services. However, children in both groups were primarily in their homes, and family preservation did not result in lower incidence of maltreatment compared with children in the control group.

Subgroups. In an effort to identify groups of cases for which family preservation is effective, subgroups of Kentucky and New Jersey cases were examined.6  Subgroups were defined in terms of problems of the family (e.g., substance abuse, financial difficulties, and depression) and family structure. Within these subgroups, experimental and control groups were compared on placement and substantiated allegations subsequent to random assignment. Only one significant difference was found. Among single mothers in New Jersey, those in the experimental group were less likely to have a subsequent substantiated allegation than those in the control group. No subgroups were found in which there were effects on placement in either state.

Family functioning. In a few areas of family functioning, in one or the other of the states, families in the experimental group appeared to be doing better at the end of services (see Table 6). There were very few differences at the year follow-up and in changes over time. Those differences that did appear (primarily at the end of services) were not consistent across states and were not maintained. Family functioning was assessed through caretaker and caregiver interviews at three points in time, shortly after the beginning of services, four to six week later (at the end of services for the Homebuilders group), and again a year after services began. Areas assessed included life events, economic functioning, household condition, child care practices, caretaker depression, child behavior, and caretaker functioning. It can be said that family preservation services may have small, apparently short-term, effects on some areas of functioning. There was one item with some consistency across sites, the overall assessment of improvement by caretakers. At post treatment, a significantly larger proportion of experimental group caretakers in Kentucky and New Jersey generally thought there was "great improvement" in their lives. In Tennessee, although not significant, results tended in the same direction.

Table 6.
Summary of family and child functioning outcomes
Differences between experimental and control groups at post treatment, follow up, and change over time
Area Post treatment Follow up (one year after beginning of treatment) Change over time
Life events
Positive life events No significant differences No significant differences No significant differences
Negative life events No significant differences No significant differences No significant differences
Depression No significant differences No significant differences No significant differences
Family problems, individual items KY: no significant differences
NJ: fewer experimentals not enough money for food, rent, or clothing
TN: fewer experimentals had few or no friends
No significant differences  
Economic functioning
Individual items KY: no significant differences
NJ: fewer experimentals difficulty paying rent and buying clothes
TN: no significant differences
KY: no significant differences
NJ: no significant differences
TN: fewer experimentals difficulty paying rent
 
Scale KY: no significant difference
NJ: experimental average lower (better)
TN: no significant difference
No significant differences No significant differences
Household condition
Individual items KY: experimentals fewer broken windows or doors
NJ: no significant differences
TN: more experimentals in unsafe building because of illegal acts
No significant differences  
Scale No significant differences No significant differences No significant differences
Area Post treatment Follow up (one year after beginning of treatment) Change over time
Child care practices
Individual items KY: fewer experimentals used punishment for not finishing food
NJ: experimentals less often got out of control when punishing child and more often encouraged child to read a book
TN: more experimentals went to amusement park, pool, or picnic
No significant differences  
Positive scale No significant differences No significant differences No significant differences
Negative scale KY: no significant difference
NJ: experimentals lower (better)
TN: no significant difference
No significant differences No significant differences
Punishment KY: no significant difference
NJ: experimentals lower (better)
TN: no significant difference
No significant differences No significant differences
Caretaker depression No significant differences No significant differences No significant differences
Child behavior
Aggression No significant differences No significant differences No significant differences
School problems No significant differences No significant differences No significant differences
Positive child behaviors No significant differences No significant differences No significant differences
Negative child behaviors KY: no significant differences
NJ: experimental group lower (better)
TN: no significant differences
No significant differences No significant differences
Overall assessment of improvement KY: experimentals, greater improvement
NJ: experimentals, greater improvement
TN: no significant difference
No significant differences  
Caseworker report of caretaker functioning
Individual items KY: no significant difference
NJ: control group higher (better) in ability in giving affection and providing learning opportunities
TN: experimental group higher (better) on five items
  KY: respecting child's opinions: experimental group declined, control group increased
NJ: control group had more positive change in respecting child's opinions 
TN: experimental group more positive change on setting firm and consistent limits
Scale KY: no significant difference
NJ: no significant difference
TN: experimental group higher (better)
  No significant differences
Caseworker report of household condition KY: control group better
NJ: control group better
TN: no significant difference
  No significant differences
Caseworker report of caretaker problems KY: experimentals more problems
NJ: no significant difference
TN: no significant difference
  KY: no significant difference
NJ: no significant difference
TN: experimentals declined more
Caseworker report of child problems No significant differences   No significant differences

5. It should be noted that the most rigorous approach to analysis requires that cases be maintained in the groups to which they were randomly assigned. Random assignment is used to assure that the groups are as similar as possible at the outset of service. Removing cases from the groups or switching cases from one group to another threatens group equality and allows for the possibility that post-treatment differences could be explained by factors other than service. In particular, it is likely that violations and minimal service cases differ in systematic ways from other cases. Hence, the secondary analyses should be viewed with caution.

6. The number of cases in Tennessee was too small to allow subgroup analysis.