Overflow designs, in which a comparison group is composed of cases not served because programs are full, provide information about effects that is somewhat better than single group or non- comparable group designs. We review four such studies here.
FamiliesFirst in Davis, California, was an intensive, in- home service program based on the Homebuilders model (Wood, Barton, and Schroeder 1988). Families were referred to the project by child protective services staff. Eligible families had children who had been abused or neglected and were thought to be at risk of having at least one child placed out of the home. An overflow comparison study was conducted. One year after intake, 25 percent (15) of the 59 children in the in-home services group were placed compared with 53 percent (26) of 49 children in the comparison group (a statistically significant difference).
Family Preservation Services in Hennepin County, consisted of intensive home-based services delivered by eight "specially trained social workers" (Schwartz and AuClaire 1989; Schwartz, AuClaire, and Harris 1991). The service was intended to last for four weeks. The evaluation of this program involved a non-random comparison group. There were 58 cases in each group, selected during the period August through December 1985. Three of the experimental group cases were in placement during the entire follow-up period and were excluded from outcome analyses. Follow-up extended until December 31, 1986. Placement occurred in 56% of 55 experimental cases and 91% of the 58 comparison cases (a significant difference). Fifty-five percent of cases in the family preservation group and 64 percent of those in the comparison group experienced multiple placements.
The Bronx Homebuilders Program, modeled after Homebuilders, began accepting clients in May 1987 (Mitchell, Tovar, and Knitzer 1989). Cases were referred from two sources, the city Child Welfare Administration (CWA) and the Pius XII Court Designated Assessment Service (Pius). The average length of service was 35 days. A one year follow-up of 45 families referred in the first year was conducted. An overflow comparison group of 12 families was available for the Pius group, one of which was not followed up. Families in the overflow group had relatively fewer placements than those in the service group. At three months, 19 percent (4 of 21) CWA, 23 percent (5 of 22) Pius treatment, and 9 percent (1 of 11) Pius comparison families had experienced a placement. At 12 months, 24 percent (5) of the CWA, 27 percent (6) of the Pius treatment, and 18 percent (2) of the Pius comparison families had experienced placement. Apparently, all children who were placed were still in placement at the end of the follow-up period.
The Family-Based Intensive Treatment (FIT) Study concerned intensive home-based services based on the Homebuilders model in Utah and Washington State (Pecora, Fraser, and Haapala 1992). In Utah a 60-day service model was provided in two sites by the state child welfare department while in Washington a 30-day service was provided in four sites by Homebuilders (under contract with the state agency). The criteria for referral were risk of imminent placement, safety of the child with service, and willingness of at least one parent to cooperate with service. At termination, 9 percent of 172 Utah children and 6 percent of 409 Washington children in the treatment groups had been placed. At a 12 month follow-up, 41 percent of 97 Utah children and 30 percent of 245 Washington children had been placed. In an overflow comparison group of 27 Utah children, 85 percent were placed during the 12 month follow-up period.13
Unfortunately, 54 percent of the cases served in the Washington project during the study period did not participate in the study.14 In addition, 32 percent of the cases in the overflow comparison group were not tracked. The researchers attempted to deal with this problem by matching a subsample of program cases with cases in the overflow comparison group; however, the variables used in the matching design were only weakly related to placement and the number of matched pairs was quite small.
Missing data for the overflow comparison group seriously compromises the interpretation of differences. If few of the unstudied overflow cases were placed, between-group differences in placement rates would have been much smaller than the differences observed.15
13 The overflow group consisted of 26 of the 38 families that were referred to the family preservation program but not served because program staff had full caseloads. They received traditional child welfare or mental health services. Twelve of the 38 families were referred to the program early on and could not be traced. The remaining 26 cases were tracked for one year or until a child at risk was placed, whichever came first (Pecora, Fraser, and Haapala 1991).
14 Of the cases that did not participate, slightly more than half (51%) were asked not to participate by their worker (for reasons that are not entirely clear), 24 percent refused to participate, 20 percent did not have the opportunity to participate because of research administration problems, and 5 percent were excluded for treatment reasons (Pecora, Fraser, and Haapala 1991).
15 Placed children in the overflow group may have been easier to track than unplaced children. If this was the case, the observed placement rate in the overflow group would have been biased upward.