Evaluation of Family Preservation and Reunification Programs: Final Report - Volume Two. 3.1 Substitute Care Placement Following Random Assignment

09/01/2002

A principal goal of family preservation services is the prevention of placement into substitute care, so that must be the first (though not the last) outcome examined. Placement included foster care, institutions and residential treatment programs, group homes, and adoptive placements. (23) We are initially concerned with the character and timing of the first placement of a child following random assignment. We collected data on placement prior to November 30, 1999 in Kentucky, September 30, 1999 in New Jersey, and August 31, 1999 in Tennessee. (24) Although data were provided at the individual level, most of the analyses are presented at the family level. (25) In Kentucky, the administrative files contained data on 1130 children in 345 families, 172 in the experimental group and 171 families in the control group. One hundred thirty-nine children in 61 families (36%) in the experimental group experienced placement subsequent to random assignment compared to 96 children in 55 families (32%) in the control group. In New Jersey, administrative data were available on 1290 children in 442 families, 275 in the experimental group and 167 in the control group. One hundred sixty-six children in 109 families (40%) in the experimental group were placed compared to 55 children in 48 families (29%) in the control group.

In Tennessee, multiple sources of data were used to calculate the rate of subsequent placement. A statewide management information system (CORS) provided information on formal paid placements. Additionally, case record reviews provided information on unpaid relative placements. In Tennessee, placement data were available on 468 children in 140 families, 93 in the experimental group and 47 in the control group. In the analysis of CORS data, forty-six children in 23 families (25%) in the experimental group experienced placement subsequent to random assignment compared to 25 children in 10 families (21%) in the control group. Including unpaid relative placements, 60 children in 29 families (31%) in the experimental group experienced placement subsequent to random assignment compared to 31 children in 13 families (28%) in the control group. These differences were not statistically significant at the family level in Kentucky, New Jersey or Tennessee (see Table 3-1 for types of placements after random assignment).

A comparison of these percents is, however, misleading, because of varying periods of risk of placement. The proper approach to the analysis of such data is survival analysis, in which the proportions of cases placed at each point in time following random assignment in each group are compared, accounting for the numbers of cases that "survive" to that point. We examined survival curves for each group and determined whether these curves were statistically different. Family level analyses were based on the first date of placement of any child in the family if a placement occurred.

Table 3-1
Type Of First Placement After Random Assignment, Child Level
Kentucky
Type N %
Foster care 144 64.0
Private institution 69 30.7
Foster care, medically fragile 6 2.7
Child psychiatric hospital 4 1.8
Not specified 2 0.8
Total 225 100
New Jersey
Type N %
Foster care 102 46.1
Juvenile family crisis 47 21.2
Residential treatment 35 15.9
Group home 17 7.7
Public institution 8 3.6
Shelter care 5 2.3
Adoptive 4 1.8
Relative 3 1.4
Total 221 100
Tennessee
Type N %
Foster care 31 44.3
Relative home 9 12.9
Trial home 6 8.5
Residential 6 8.5
Continuum contract 4 5.7
Non-relative home 4 5.7
Adoptive home 3 4.3
Runaway 2 2.8
Shelter 2 2.8
Independent living 2 2.8
Detention 1 1.4
Total 71 100
Note: Includes only placements recorded in administrative data. There were additional unpaid relative placements (see text).

Kentucky. The family level analysis of subsequent placement is displayed in Figure 3-1. (26) These survival curves show the proportion of families remaining intact (without placement of a child) at each point in time following random assignment. The curves begin at 1, indicating that at the time of random assignment, all children were at home. The curves then decline as children enter care. The higher curve at any point represents the group with fewer placed children at that point. The curves are adjusted for cases that are "right censored." For example, cases that were not observed for a full year following random assignment are dropped in the calculation of the percentage remaining intact ("surviving") at one year. The Wilcoxon statistic indicates that the survival rates for the experimental and control groups are not statistically different. At the one-year interval, 25 percent of experimental group families and 24 percent of control group families experienced substitute care placement. At the end of two years, 32 percent of the experimental group and 27 percent of the control group families experienced substitute care placement.

"Refined" groups analyses were also conducted, limiting the sample to cases referred by investigative workers and to those families with substantiated allegations within the three months prior to random assignment. Significant differences did emerge for families with a substantiated allegation within three months prior to random assignment. In the primary analysis of those families coming from an investigative worker, 26 percent of the experimental group and 15 percent of the control group experienced subsequent placement within one year after the random assignment date. For those with recent substantiated allegations, 29 percent of the experimental group and 13 percent of the control group experienced subsequent placement within one year (significant at .05 level).

An additional "refined" group was available for analysis in Kentucky. Prior to random assignment, workers submitted petitions to the court for placement or some other court ordered intervention on 66 families. Administrative data were available for all 66 families (32 in the experimental group, 34 in the control group). Survival analyses were conducted to explore the relationship between family preservation services and subsequent placement. At one year after random assignment, 22 percent of the experimental group and 29 percent of the control group experienced placement; a nonsignificant difference.

Figure 3-1
First Placement after Random Assignment (Families)

Figure 3-1 First Placement after Random Assignment (Families) (Kentucky)Figure 3-1 First Placement after Random Assignment (Families) (New Jersey)

Figure 3-1 First Placement after Random Assignment (Families) (Tennesse, CORS Administrative Data)Figure 3-1 First Placement after Random Assignment (Families) (Tenessee, Any Evidence)

In addition to the administrative data on placement, in Kentucky the Westat site coordinator attempted to document all placements subsequent to random assignment, based on her contacts with caseworkers. The administrative data file contained placements not recorded by the site coordinator, and vice versa. The only systematic difference between these data sources was the documentation of relative placement. Relatives are generally not paid for placements in Kentucky, so these data were not recorded in the administrative files. Survival analyses were conducted with a combination of caseworker and administrative placement records. If either data source recorded a placement event, that family was coded as experiencing subsequent placement. The first documented date of placement, taken from either source, was selected for analysis. The patterns of placement in these analyses are similar to those reported above. At one year, 27 percent of the experimental group and 32 percent of the control group families experienced placement, a nonsignificant difference.

In addition to survival analyses, placement can be examined in terms of the proportion of time in substitute care subsequent to random assignment. If family preservation services are effective in preventing placements, we would expect them to result in lower numbers of days in foster care. Family preservation might also result in shorter stays in care, once children are placed. Comparison of days in care provides a beginning look at the question of whether family preservation results in lower costs of foster care (of course, a complete cost-effectiveness analysis must also factor in the differential costs of family preservation and regular services).

The proportion of time in care is calculated by dividing the number of days in care by the number of days of possible care (number of days between random assignment and the date of administrative data collection). As the proportions are calculated at the family level, the number of days in care represents the total number of care days summed across all children within a particular family. Similarly, the number of possible care days represents the total number of possible care days summed across all children within a particular family. The number of possible care days is adjusted for a child's eighteenth birthday and for births since random assignment. For both primary and secondary analyses, in both the experimental and control groups children spent an average of 6 percent of the days subsequent to random assignment in care.

New Jersey. The family level analysis of placements is shown in Figure 3-1. (27) More families in the experimental group experienced placement of a child than in the control group (at one year, 29% of the experimental group vs. 22 percent of the control group; at two years, 39 percent of the experimental group vs. 28 percent of the control group) although the differences are not significant. It might be noted that in the analyses the survival curves for the two groups tend to begin to diverge at about 6-7 months, that is, at about that time more children in the experimental group are being placed. We do not have a ready explanation for this divergence.

Refined groups analyses in New Jersey revealed statistically significant differences. In the primary analysis of those families coming from an investigative worker, 25 percent of the experimental group and 15 percent of the control group experienced subsequent placement within one year of the random assignment date. For those with recent substantiated allegations, 25 percent of the experimental group and 14 percent of the control group experienced subsequent placement.

As to the proportion of time that children spent in care in New Jersey, experimental group children spent an average of 6 percent of that time in placement, compared to 5 percent for the control group children (not a significant difference).

Tennessee. Survival rates at the family level were first calculated using only the CORS and then including relative placement (the "any evidence" analysis) data. The family level analyses of subsequent placement is displayed in Figure 3-1. (28) The Wilcoxon statistic indicates that the survival rates for the experimental and control groups are not statistically different. In the analysis of CORS data, 23 percent of experimental group families and 19 percent of control group families experienced substitute care placement within one year subsequent to random assignment. In the "any evidence" analysis, 28 percent of the experimental group families and 23 percent of control group families experienced placement within one year subsequent to random assignment.

As in Kentucky and New Jersey, a "refined" group was available for analysis in Tennessee. Ninety-three families had an allegation within 30 days prior to random assignment. The Wilcoxon statistic for the survival analysis of placement in these families indicates that the survival rates of the two groups are not statistically different. In the analysis of CORS data, 17 percent of the experimental group and 15 percent of the control group experienced subsequent placement within one year of random assignment. In the "any evidence" analysis, 22 percent of the experimental group and 21 percent of the control group experienced subsequent placement within one year of random assignment.

As to the proportion of time that children spent in care in Tennessee, experimental group children spent an average of 10 percent of that time in placement, compared to 5 percent for the control group children. This difference is nonsignificant.

In a number of analyses of subsequent placement in these states, more experimental group families experienced placement than did control group families. In a few analyses, fewer experimental group families experienced placement. However, none of these analyses were statistically significant; in none of these states can the data be taken as firm evidence that family preservation resulted in more placements. Nor is there evidence that it resulted in fewer.

Imminent Risk of Placement. The family preservation programs in these states are designed to prevent the unnecessary removal of children by serving families with children who are at imminent risk of out-of-home placement. (29) One way to explore the accuracy of the "imminent risk" designation is to examine the proportion of control group families that experienced placement within a short time after random assignment. Since the control and experimental groups were randomly assigned and are expected to be statistically equivalent before services are begun, the proportion of families experiencing placement in the control group indicates the proportion of referred families that would have experienced placement in the absence of receiving family preservation services. We looked at control group placement rates 30 days after random assignment, believing that time period provided a liberal interpretation of "imminent risk." If a significant proportion of the control group experienced placement within 30 days of random assignment, one could argue that the program was appropriately targeted. At the time of random assignment, referring workers were asked to designate those children who were considered "at risk."

In Kentucky, in the first 30 days following random assignment, in the primary analysis 4 percent of at risk children in the experimental group were placed compared to 3 percent of control group at risk children. At the family level, 6 percent of the experimental group families and 5 percent of the control group families experienced placement within the first 30 days subsequent to random assignment. The percentages were similar in the investigative group (8% of the experimental compared with 5% of the control group), and among those with recent substantiated allegations (6% of the experimental group compared with 3% of the control group).

In New Jersey, of those children judged to be at risk, 4 percent of the control group and 3 percent of the experimental group were placed in 30 days. At the family level, 5 percent of the families in the experimental group experienced placement of at least one child within one month of random assignment, compared to 6 percent of the control group. Rates of imminent placement were similar in the "refined" group analyses. Of those families coming from an investigative worker, 3 percent of the experimental group and 5 percent of the control group experienced placement within 30 days of random assignment. For those families with a substantiated allegation within three months prior to random assignment, 8 percent of the experimental group and 5 percent of the control group experienced placement within 30 days.

In Tennessee, rates of placement within one month were somewhat higher than in Kentucky and New Jersey. Of those children judged to be at risk, 13 percent of the control group and 11 percent of the experimental group were placed in 30 days. There were no relative placements within the first 30 days subsequent to random assignment. Thus, there are no differences between the CORS and "any evidence" analysis. At the family level in Tennessee, the CORS administrative data indicates that 11 percent of both the experimental and control groups experience placement within 30 days subsequent to random assignment. Rates of imminent placement were similar in the "refined" group analyses. Of those families with a recent allegation (within 30 days prior to random assignment), 7 percent of the experimental group and 12 percent of the control group experienced a CORS placement within 30 days of random assignment.

Although the percentages of placement within one month were somewhat higher in Tennessee, in all three states, these percentages were quite low. The numbers of interest here are those for the control group, indicating the targeting efficiency of the program in these three sites is very low.

View full report

Preview
Download

"report2.pdf" (pdf, 978.24Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®