Evaluation of Family Preservation and Reunification Programs: Interim Report. 8.1 Substitute Care Placement Following Random Assignment

01/08/2001

A principal goal of family preservation services is the prevention of placement into substitute care, so that must be the first (though not the last) outcome examined. Placement included foster care, institutions and residential treatment programs, group homes, and adoptive placements.1 We are initially concerned with the character and timing of the first placement of a child following random assignment. We collected data on placement prior to August 31, 1998 in Kentucky and New Jersey and before August 31, 1999 in Tennessee.2 Although data were provided at the individual level, most of the analyses are presented at the family level.3 In Kentucky, the administrative files contained data on 942 children in 306 families, 154 in the experimental group and 152 families in the control group. Eighty-six children in 40 families (26%) in the experimental group experienced placement subsequent to random assignment compared to 60 children in 41 families (27%) in the control group. In New Jersey, administrative data were available on 1230 children in 434 families, 269 in the experimental group and 165 in the control group. One hundred sixteen children in 81 families (30%) in the experimental group were placed compared to 44 children in 39 families (24%) in the control group. In Tennessee, multiple sources of data were used to calculate the rate of subsequent placement. A statewide management information system (CORS) provided information on formal paid placements. Additionally, case record reviews provided information on unpaid relative placements. In Tennessee, placement data were available on 468 children in 140 families, 93 in the experimental group and 47 in the control group. In the analysis of CORS data, forty-six children in 23 families (25%) in the experimental group experienced placement subsequent to random assignment compared to 25 children in 10 families (21%) in the control group. Including unpaid relative placements, 60 children in 29 families (31%) in the experimental group experienced placement subsequent to random assignment compared to 31 children in 13 families (28%) in the control group. These differences were not statistically significant at the family level in Kentucky, New Jersey or Tennessee (see Table 8-1 for types of placements after random assignment).

A comparison of these percents is, however, misleading, because of varying periods of risk of placement. The proper approach to the analysis of such data is survival analysis, in which the proportions of cases placed at each point in time following random assignment in each group are compared, accounting for the numbers of cases that "survive" to that point. We examined survival curves for each group and determined whether these curves were statistically different. Family level analyses were based on the first date of placement of any child in the family if a placement occurred.

Table 8-1.
Type of first placement after random assignment, child level
Kentucky
Type N Percentage
Foster care 96 66
Private institution 42 29
Foster care, medically fragile 4 2.1
Child psychiatric hospital 3 2.7
Not specified 1 0.7
Total 146 100
New Jersey
Foster care 76 47.5
Juvenile family crisis 33 20.6
Residential treatment 25 15.6
Group home 13 8.1
Shelter care 4 2.5
Public institution 4 2.5
Relative 3 1.9
Independent living 3 0.6
Maternity home 1 0.6
Total 160 100
Tennessee
Foster care 31 44.3
Relative home 9 12.9
Trial home 6 8.5
Residential 6 8.5
Continuum contract 4 5.7
Non-relative home 4 5.7
Adoptive home 3 4.3
Runaway 2 2.8
Shelter 2 2.8
Independent living 2 2.8
Detention 1 1.4
Total 71 100

Note: Includes only placements recorded in administrative data. There were additional unpaid relative placements (see text).


Kentucky. The family level analysis of subsequent placement is displayed in Figure 8-1.4  These survival curves show the proportion of families remaining intact (without placement of a child) at each point in time following random assignment. The curves begin at 1, indicating that at the time of random assignment, all children were at home. The curves then decline as children enter care. The higher curve at any point represents the group with fewer placed children at that point. The curves are adjusted for cases that are "right censored." For example, cases that were not observed for a full year following random assignment are dropped in the calculation of the percentage remaining intact ("surviving") at one year. The Wilcoxon statistic indicates that the survival rates for the experimental and control groups are not statistically different. At the one-year interval, 23 percent of experimental group families and 24 percent of control group families experienced substitute care placement.

Figure 8-1 First Placement after random assignment (families)

Figure 8-1_1 First Placement after random assignment (families)Figure 8-1_2 First Placement after random assignment (families)Figure 8-1_3 First Placement after random assignment (families)Figure 8-1_4 First Placement after random assignment (families)

"Refined" groups analyses were also conducted, limiting the sample to cases referred by investigative workers and to those families with substantiated allegations within the three months prior to random assignment. No statistically significant differences emerged. In the primary analysis of those families coming from an investigative worker, 26 percent of the experimental group and 15 percent of the control group experienced subsequent placement within one year after the random assignment date. For those with recent substantiated allegations, 29 percent of the experimental group and 16 percent of the control group experienced subsequent placement within one year.

An additional "refined" group was available for analysis in Kentucky. Prior to random assignment, workers submitted petitions to the court for placement or some other court ordered intervention on 67 families. Administrative data were available for 60 of these 67 families (29 in the experimental group, 31 in the control group). Survival analyses were conducted to explore the relationship between family preservation services and subsequent placement. At one year after random assignment, 18 percent of the experimental group and 33 percent of the control group experienced placement; a nonsignificant difference.

In addition to the administrative data on placement, in Kentucky the Westat site coordinator attempted to document all placements subsequent to random assignment, based on her contacts with caseworkers. The administrative data file contained placements not recorded by the site coordinator, and vice versa. The only systematic difference between these data sources was the documentation of relative placement. Relatives are generally not paid for placements in Kentucky, so these data were not recorded in the administrative files. Survival analyses were conducted with a combination of caseworker and administrative placement records. If either data source recorded a placement event, that family was coded as experiencing subsequent placement. The first documented date of placement, taken from either source, was selected for analysis. The patterns of placement in these analyses are similar to those reported above. At one year, 27 percent of the experimental group and 32 percent of the control group families experienced placement, a nonsignificant difference.

In addition to survival analyses, placement can be examined in terms of the proportion of time in substitute care subsequent to random assignment. If family preservation services are effective in preventing placements, we would expect them to result in lower numbers of days in foster care. Family preservation might also result in shorter stays in care, once children are placed. Comparison of days in care provides a beginning look at the question of whether family preservation results in lower costs of foster care (of course, a complete cost-effectiveness analysis must also factor in the differential costs of family preservation and regular services).

Table 8-2.
Subgroup analyses, significance levels of differences between experimental and control groups
  Kentucky New Jersey
Placement Subtantiated allegations Placement Subtantiated allegations
In 6 mos In 6 mos In 6 mos In 6 mos
N Overall % pa Survivalp Overall % pa Survivalp N Overall % pa Survivalp Overall % pa Survivalp
Overall 306 18     14     434 18     6    
Substance abuse 28 25   .099b 25     50 12     16 .10  
No substance abuse 231 17     13     322 20     4    
Problems with bills 137 20     19     192 17     6    
Problems with daycare 92 19     14     111 22     8    
Depressiond 149 20     16     165 23     7    
Problems with punishment 187 19     15     256 20     5    
Problems with school 134 14     16     196 22     5    
Problems with employment 172 17     16     189 16     6    
Single mother 118 18     14     110 16   .05 6    

a Fisher exact, two tail

b Experimental group more likely to experience placement

c Control group more likely to experience placement

d Caretakers with depression scores above median for the state


The proportion of time in care is calculated by dividing the number of days in care by the number of days of possible care (number of days between random assignment and the date of administrative data collection). As the proportions are calculated at the family level, the number of days in care represents the total number of care days summed across all children within a particular family. Similarly, the number of possible care days represents the total number of possible care days summed across all children within a particular family. The number of possible care days is adjusted for a child's eighteenth birthday and for births since random assignment. For both primary and secondary analyses, in both the experimental and control groups children spent an average of 6 percent of the days subsequent to random assignment in care.

New Jersey. The family level analysis of placements is shown in Figure 8-1.5 More families in the experimental group experienced placement of a child than in the control group (at one year, 28% of the experimental group vs. 22% of the control group) although the differences are not significant. It might be noted that in the analyses the survival curves for the two groups tend to begin to diverge at about 2-3 months, that is, at about that time more children in the experimental group are being placed. We do not have a ready explanation for this divergence.

Refined groups analyses in New Jersey revealed no statistically significant differences. In the primary analysis of those families coming from an investigative worker, 25 percent of the experimental group and 16 percent of the control group experienced subsequent placement within one year of the random assignment date. For those with recent substantiated allegations, 27 percent of the experimental group and 15 percent of the control group experienced subsequent placement.

As to the proportion of time that children spent in care in New Jersey, experimental group children spent an average of 6 percent of that time in placement, compared to 4 percent for the control group children (not a significant difference).

Tennessee. Survival rates at the family level were first calculated using only the CORS and then including relative placement (the "any evidence" analysis) data. The family level analyses of subsequent placement is displayed in Figure 8-1.6 The Wilcoxon statistic indicates that the survival rates for the experimental and control groups are not statistically different. In the analysis of CORS data, 23 percent of experimental group families and 19 percent of control group families experienced substitute care placement within one year subsequent to random assignment. In the "any evidence" analysis, 28 percent of the experimental group families and 23 percent of control group families experienced placement within one year subsequent to random assignment.

As in Kentucky and New Jersey, a "refined" group was available for analysis in Tennessee. Ninety-three families had an allegation within 30 days prior to random assignment. The Wilcoxon statistic for the survival analysis of placement in these families indicates that the survival rates of the two groups are not statistically different. In the analysis of CORS data, 17 percent of the experimental group and 15 percent of the control group experienced subsequent placement within one year of random assignment. In the "any evidence" analysis, 22 percent of the experimental group and 21 percent of the control group experienced subsequent placement within one year of random assignment.

As to the proportion of time that children spent in care in Tennessee, experimental group children spent an average of 10 percent of that time in placement, compared to 5 percent for the control group children. This difference is nonsignificant.

In a number of analyses of subsequent placement in these states, more experimental group families experienced placement than did control group families. In a few analyses, fewer experimental group families experienced placement. However, none of these analyses were statistically significant; in none of these states can the data be taken as firm evidence that family preservation resulted in more placements. Nor is there evidence that it resulted in fewer.

Imminent risk of placement. The family preservation programs in these states are designed to prevent the unnecessary removal of children by serving families with children who are at imminent risk of out-of-home placement.7 One way to explore the accuracy of the "imminent risk" designation is to examine the proportion of control group families that experienced placement within a short time after random assignment. Since the control and experimental groups were randomly assigned and are expected to be statistically equivalent before services are begun, the proportion of families experiencing placement in the control group indicates the proportion of referred families that would have experienced placement in the absence of receiving family preservation services. We looked at control group placement rates 30 days after random assignment, believing that time period provided a liberal interpretation of "imminent risk." If a significant proportion of the control group experienced placement within 30 days of random assignment, one could argue that the program was appropriately targeted. At the time of random assignment, referring workers were asked to designate those children who were considered "at risk."

In Kentucky, in the first 30 days following random assignment, in the primary analysis 4 percent of at risk children in the experimental group were placed compared to 3 percent of control group at risk children. At the family level, 6 percent of the experimental group families and 5 percent of the control group families experienced placement within the first 30 days subsequent to random assignment. The percentages were similar in the investigative group (8 percent of the experimental compared with 5 percent of the control group), and among those with recent substantiated allegations (6 percent of the experimental group compared with 3 percent of the control group).

In New Jersey, of those children judged to be at risk, 4 percent of the control group and 3 percent of the experimental group were placed in 30 days. At the family level, 4 percent percent of the families in the experimental group experienced placement of at least one child within one month of random assignment, compared to 6 percent of the control group. Rates of imminent placement were similar in the "refined" group analyses. Of those families coming from an investigative worker, 3 percent of the experimental group and 5 percent of the control group experienced placement within 30 days of random assignment. For those families with a substantiated allegation within three months prior to random assignment, 8 percent of the experimental group and 5 percent of the control group experienced placement within 30 days.

In Tennessee, rates of placement within one month were somewhat higher than in Kentucky and New Jersey. Of those children judged to be at risk, 13 percent of the control group and 11 percent of the experimental group were placed in 30 days. There were no relative placements within the first 30 days subsequent to random assignment. Thus, there are no differences between the CORS and "any evidence" analysis. At the family level in Tennessee, the CORS administrative data indicates that 11 percent of both the experimental and control groups experience placement within 30 days subsequent to random assignment. Rates of imminent placement were similar in the "refined" group analyses. Of those families with a recent allegation (within 30 days prior to random assignment), 7 percent of the experimental group and 12 percent of the control group experienced a CORS placement within 30 days of random assignment.

Although the percentages of placement within one month were somewhat higher in Tennessee, in all three states, these percentages were quite low. The numbers of interest here are those for the control group, indicating the targeting efficiency of the program in these three sites is very low.


(1)  The full list of New Jersey service codes that were included is: public institution, teaching family placement, para-foster care income maintenance, juvenile-family crisis shelter placement, relative placement, foster care placement, residential treatment placement, finalized adoption placement, selected adoption placement -- pending, maternity home care, group home placement, independent living, and shelter care placement. Four of these categories did not actually occur in the data: teaching family placement, para-foster care income maintenance, finalized adoption placement, and selected adoption placement -- pending. In Kentucky placement (as reflected in the variable FACTYPE), included: adoption, foster care, private institution/boarding schools, family treatment home, unmarried parent, other, children's psychiatric hospital, and foster care medically fragile. The data did not include adoption, family treatment home, and unmarried parent. In Tennessee, placements included: foster care, relative home, trial home, residential care, continuum contract, non-relative home, adoptive home, runaway, shelter, independent living, and detention.

(2)  Cases entered the study at varying points in time. In Kentucky, cases entered between May 7, 1996 and February 13, 1998; in New Jersey, cases entered between November 6, 1996 and February 26, 1998; and in Tennessee, between November 19, 1996 and May 26, 1998.

(3)  There are two reasons for focusing on family level analyses. First, we are not confident that the administrative data allow for accurate identification of children to be included in the risk pool (what would be the denominator in a rate of placement calculation). Children are identified as belonging to a family through a case number. The analysis requires that we identify children who are in the home at the time of random assignment (or who are born or return to the home subsequently). In these states, children apparently often retain a family case number even when they are not in the home, and the administrative data do not allow us to verify the location of the child at the time of random assignment (or even sometimes at the time of an event such as placement). This problem is alleviated in analyses at the family level, since we know that the family is at risk of having a child placed (as long as there are any children in the family).

As to the accuracy of the "numerator" in our analyses, we focus on the first event (e.g., placement) in the family, subsequent to random assignment. It is possible that the first event occurs with regard to a child identified with a family but not living in that family at the time of the event. We judge the likelihood of that occurring to be small (the effects of this source of error would be similar in a family and child level analysis). In addition, subsequent events involving other children identified with the family but not in the family at the time of the event would not affect the family level analysis, while they would create inaccuracies in a child level analysis.

The second reason for focusing on the family level has to do with a "clustering" effect in the child level analysis. Clustering refers to the lack of independence between children within the same family of observations of such things as placement. If one child is removed from the home, the remaining children are more likely to experience placement. The "clustering effect" leads to an underestimate of the significance levels when analyses are conducted at the child level. Conducting the analyses at the family level is one approach to resolving this dilemma.

We did conduct a few analyses at the child level, when we wanted to take into account child characteristics, but it should be remembered that significance levels in those analyses are downwardly biased.

(4)  In Kentucky, the ratio of assignment to experimental and control groups was 50-50.

(5)  In New Jersey, approximately 60% of the cases were assigned to the experimental group.

(6)  In Tennessee, approximately two-thirds of the cases were assigned to the experimental group.

(7)  Kentucky policy specifies that imminent risk includes children who are at risk of commitment as dependent, abused, or neglected; who are identified through the Regional Interagency Council, an interdepartmental unit, as severely emotionally disturbed; or whose families are in conflict such that they are unable to exercise reasonable control of the child. Both the referring worker and family members shall believe that without immediate intensive intervention, out-of-home placement is imminent. At the time of this study, New Jersey targeted family preservation services for families at imminent risk of having at least one child enter placement. The referring worker must have based the assessment of imminent risk on a face-to-face interview with the family no more than 5 days prior to the referral. The family must need services immediately and the worker must determine that other, less intensive, services have been used, are not appropriate, or are not available. In Tennessee, CPS intake workers complete a risk assessment form to identify high, intermediate, low, or no risk. High risk cases are identified as cases where "the child or children in the home are at imminent risk of serious harm if there is no intervention in the situation." A typical high risk case might involve such factors as: 1) a vulnerable child; 2) a history of previous maltreatment; 3) an active perpetrator who has continued access to the child; and 4) no available support or family strengths to offset the stated risks.