Evaluation of Family Preservation and Reunification Programs: Final Report

12/01/2002

Contents

Endnotes

Background

This report presents an evaluation of family preservation programs. Family preservation programs are intended to prevent the placement of children in foster care when it can be avoided.(1) This report focuses on programs in four states. Three of the sites employ the Homebuilders model of family preservation, thought by many to be the most promising approach. The fourth site employs a broader, home-based, family preservation service model.

An interim evaluation report was released in October 2000. The interim report presented description, service, and outcome analyses on the Homebuilders study sites. This report expands on the interim report by including description, service, and outcome analyses of the non-Homebuilders site. Additionally, analyses on sample attrition, social support, investigating worker questionnaires, staff questionnaires, and secondary analyses are included in this report.(2)

Society has accepted a measure of responsibility for the well-being of children. These measures allow government to intervene in family life when a child is severely threatened by abuse or neglect, dependency due to death or disability of parents, or family conflict. Governmental intervention includes removing children from their homes when that is necessary. However, it has long been thought that children should remain in their parent's care whenever possible, consistent with their safety. The tension between assuring the safety of children and maintaining the integrity of families has been a perennial source of debate in the child welfare field and in our society more generally.

Legislation

In 1980, Congress passed the Adoption Assistance and Child Welfare Act of 1980 (P.L. 96-272). This Act required states to make "reasonable efforts" to prevent children from entering foster care and to return children who are in foster care to their families. Part of the response of states to that Act was the development of family preservation programs. The emphasis on family preservation was further codified in the 1993 Omnibus Budget Reconciliation Act, which established a 5-year capped entitlement program to encourage the development of family preservation and family support programs.

This program was revised and extended by P.L. 105-89, the 1997 Adoption and Safe Families Act (ASFA). The Adoption and Safe Families Act changed and clarified a number of policies established in the 1980 Act with a renewed emphasis on safety, permanency, and adoption. ASFA placed Federal family preservation initiatives under the rubric of "Promoting Safe and Stable Families" and extended funding for FY 2001. The law made safety of children the paramount concern in service delivery and increased the need to understand how family preservation services strengthen families and prevent foster care placement and subsequent abuse and neglect allegations.

Public Law 107-133, the "Promoting Safe and Stable Families Amendments of 2001" was signed into law in January 2002. This legislation reauthorized family preservation services through 2006. Additionally, the legislation authorized the Court Improvement Program, and offered states flexibility in defining family preservation services to allow states to support infant safe haven programs and strengthen parental relationships and promote healthy marriages.

Evaluations

There have been a number of other evaluations of family preservation programs. Early evaluations suggested these programs had considerable promise but these studies were criticized for flaws in research design. Later, more rigorously designed studies began to cast doubt on the extensive claims of success. The largest of these studies were in California, New Jersey, and Illinois. No placement prevention effects were found in California and Illinois, while the study in New Jersey found short-term effects that dissipated with time.(3) However, these studies were also criticized, most notably for not having examined programs thought to be most effective, those based on the Homebuilders approach.

The evaluation reported here was mandated by Congress in the 1993 legislation and was intended, in part, to provide information for deliberations on reauthorization of the funding. It is hoped that the evaluation will also be useful to the states in making decisions about child welfare programs and to program planners and practitioners in developing responses to significant social problems.

The evaluation was designed to overcome shortcomings of previous studies of family preservation programs. It studied the Homebuilders model of service in the states of Kentucky, New Jersey, and Tennessee. The Homebuilders model is the approach to family preservation that many observers believe to be the most effective. The evaluation also studied a program model somewhat less intensive than Homebuilders, in Philadelphia. The evaluation examined a number of outcomes. Placement prevention is a major goal of these programs, but family preservation is expected to achieve that goal while assuring the safety of children. A further important goal of these programs is improvement in functioning of parents, children, and families. Finally, it is expected that these programs will enable child welfare agencies to close cases more quickly, ending their involvement with families. Hence, besides placement prevention, the evaluation assessed the safety of children, changes in child and family functioning, and rates of case closure.

An additional issue raised in the earlier evaluations of family preservation concerned the targeting of these programs. It was found that the families served by these programs often were not those for whom they were intended: cases in which it was likely that at least one child would be placed in foster care without special intervention. The evaluation sought to throw light on this issue for the Homebuilders models as well.

The Homebuilders Model

Homebuilders, a foster care placement prevention program developed in 1974 in Tacoma, Washington, calls for short-term, time-limited services provided to the entire family in the home.(4) The program is based, in part, on crisis intervention theory. This theory holds that families experiencing a crisis  that is, about to have a child placed in foster care  will be more amenable to receiving services and learning new behaviors. Social learning theory also plays a part in defining the Homebuilders model. Social learning theory rejects the belief that changes in thinking and feeling must precede changes in behavior. Instead, behavior, beliefs, and expectations influence each other in a reciprocal manner. Key program characteristics include:

  • contact with the family within 24 hours of the crisis
  • caseload sizes of one or two families per worker
  • service duration of four to six weeks
  • provision of concrete services and counseling
  • the family receiving up to 20 hours of service per week.

Broader Home-based Family Preservation Service Model

The broader home-based model focuses on the behavior of the family overall, and attempts to change the way in which the family functions as a whole and within the community. Aside from a primary goal of placement prevention, the model also seeks to improve functioning of parents, families, and children. Programs using the home-base model stress longer-term interventions based on family systems theory. One study site, the Philadelphia Family Preservation Services (FPS), used a broader home-based model. FPS tailored home-based services to build upon the Pennsylvania Free substance abuse services provided in the 1980s. Key characteristics of the Philadelphia FPS program included: 12 weeks of service to families, focus on drug and alcohol abuse in families, caseload sizes of five families per worker, and provision of both concrete services and counseling.

[Go To Contents]

Evaluation Design

The design for this evaluation was an experiment in which families were randomly assigned either to a family preservation program (the experimental group) or to other, "regular," services of the child welfare system (the control group). This report concerns programs in Louisville, Kentucky; seven counties in New Jersey; Memphis, Tennessee; and Philadelphia, Pennsylvania. Information was collected through interviews with caseworkers and caretakers to examine caretakers' parenting practices, interaction with children, discipline, social networks, economic functioning, housing, abuse and neglect, psychological functioning, child well-being, and caseworker/caretaker interactions. These interviews were conducted with:

  • The investigating worker, caseworker, and caretaker of each family at the start of services;
  • The caseworker and the caretaker at the conclusion of family preservation services and at a comparable point in time for families in the control group; and
  • The caretaker one year after entry into the experiment.

After each in-person contact with families, experimental and control caseworkers completed a one-page form describing the services provided during the contact. Administrative data provided information on children's placements, reentries, and subsequent abuse and neglect allegations up to 18 months after entry into the experiment. Staff attitudes and characteristics were collected through a self-administered questionnaire. Throughout the project, discussions were held with personnel of the public agency and service provider agency to gather information about agency services, policies, staffing, training, and the context of services.

Site Descriptions

While data collection efforts were the same across sites, the sites varied in their approach to identifying families for services, the populations served, and the type of services provided (Table 1).

Table 1.
Study Site Descriptions
Program Description Kentucky New Jersey Tennessee Philadelphia
Program Attributes
Location of evaluation Jefferson County (Louisville) Fayette County (Lexington) Bergen, Burlington, Camden, Essex, Monmouth, Ocean, and Passaic counties. Shelby County (Memphis) Philadelphia County
Program type Statewide FP program Statewide FP program Statewide FP program County FP program
Program model Homebuilders model Homebuilders model Homebuilders model Specialized program model

Responsibility for:

Selection criteria State office coordinator State office coordinator State office coordinator Public specialized FPS section
Training State office coordinator State office coordinator State office coordinator State DHS office
FP provider oversight State office coordinator State office coordinator State office coordinator State DHS office
Providers Single FPS provider in study location. Single FPS provider in each county location. Single FPS provider in study. Three private FPS providers in study
Screener Targeted cases were at high risk and should have entered foster care without FP. High-risk family court cases where a petition was filed were reviewed for placement in the study.

Public agency screener reviewed all cases referred to FPS for appropriateness.

Targeted cases were at high risk and should have entered foster care without FP.

Each county had a screener to review cases referred for FP and make sure there were openings in the program.

Targeted cases were at high risk and would have entered foster care without FP.

For the study, the screener referred cases to the FP program (prior to the study workers referred cases directly to program)

Targeted cases were at intermediate risk of removal from home.

DHS FPS supervisor screened cases to the FPS program and determined if there was an opening in the program.

Population Attributes
Population criteria FP cases referred from intake and ongoing units. FP cases referred from intake and ongoing cases. FP cases referred from intake only. FP cases were referred from CPS intake only
Child age limit Children under 18 years of age.

At time of study, the state was trying to refocus delivery of FP to younger children.

All children under 18.

At the time of study, the state was trying to refocus delivery of FP to younger children but not all counties modified targeting.

1 child in the family had to be under 13 years of age. All children under 18

The program originally focused on young children but progressed to serving families with older children

  • Kentucky had a statewide program that uses the Homebuilders model. A state office coordinator was responsible for developing uniform selection criteria, training, contracting with family preservation providers, and overseeing the state program. The evaluation was conducted in Louisville. This location provided a single-family preservation provider agency. Child abuse and neglect cases in Louisville were referred from intake or ongoing workers. A public agency screener reviewed all cases referred for family preservation services. Her role was to ensure that cases were appropriate for the service. There was no age limitation on the children included in the experiment. In Kentucky, there were 174 cases in the experimental group and 175 in the control group.
  • New Jersey had a statewide program using the Homebuilders model at the time of the study. During the data collection, a state office coordinator was responsible for developing uniform selection criteria, training, contracting with providers, and program oversight. The study was conducted in seven counties: Bergen, Burlington, Camden, Essex, Monmouth, Ocean, and Passaic. Each county had a separate family preservation provider agency. The study population included Division of Youth and Family Service child abuse and neglect and family problem cases (primarily adolescent-parent conflict cases) referred from intake or ongoing workers. Each county had a screener to review cases referred for family preservation. Their major role was to review the appropriateness of the referrals and to make sure there were openings in the program. When the study began, the state was trying to refocus delivery of family preservation services to families with younger children. Not all counties conformed to this expectation, so all children under 18 were included in the experiment. In New Jersey, there were 275 cases in the experimental group and 167 in the control group.
  • Tennessee had a statewide program using the Homebuilders model during the study period. It also had a state office coordinator responsible for developing uniform selection criteria, training, contracting with providers, and program oversight. The evaluation was conducted in Memphis and focused on families with children under 13 years old referred from the Department of Children's Services. Cases were referred only from intake workers. Prior to the study, workers referred cases directly to the family preservation program. For the study, cases were referred to a screener rather than directly to the program. In Tennessee, there were 98 cases in the experimental group and 49 in the control group.
  • Philadelphia had a family preservation program that used a broader service model than the traditional Homebuilders model during the study period. The state office was responsible for training and program oversight. The agency-specialized FPS section developed selection criteria for referral. FPS were provided by private agencies in a public-private collaboration. The evaluation included three private agencies  Abraxas Foundation, Tabor Children's Services and Youth Service, Inc. FPS were provided by Abraxas Foundation and Tabor Children's Services. All three agencies provided non-FPS Services to Children in their Own Home (SCOH) services to families. Cases were referred only from intake workers. Referrals came through a public supervisor who screened cases for FPS. In Philadelphia, there were 209 cases in the experimental group and 144 in the control group.

[Go To Contents]

The Families

Most families in the study had birth mothers as the primary caretakers. In Kentucky, New Jersey, and Tennessee about half of these women had not graduated from high school. In Philadelphia, 65 percent of the women had not graduated from high school. Half of the households in Tennessee and Philadelphia were headed by a single-birth mother, compared to 43 percent in Kentucky, and 34 percent in New Jersey (Table 2).

 

Table 2
Description of the Families at Time of Initial Interviews
  Kentucky New Jersey Tennessee Philadelphia
N % N % N % N %
Gender of caretaker/respondent 311   328   117   263  
Male   7   12   7   5
Female   93   88   93   95
Race of caretaker/respondent 310   327   116   263  
African American (not Hispanic)   43   42   83   80
Caucasian (not Hispanic)   55   47   15   15
Hispanic   1   9   1   2
Other   1   2   0   2
Respondent's education level 311   325   116   263  
Elementary school or less   9   9.4   9   4
Some high school   44   40   46   61
High school graduate or obtained GED   32   26   18   19
College   14   20   22   11
Special education or vocational schooling   1   4.0   4   4
Respondent's marital status 310   328   117   263  
Married   24   30   17   10
Divorced   19   23   13   7
Separated   21   11   14   11
Widowed   3   6   3   3
Never married   33   30   54   69
Respondent's relationship to youngest child 292   326   117   263  
Birth mother   85   69   85   91
Biological father   67   10   6   5
Grandmother   6   12   3   3
Other relative   2   9   6   2
Household composition 311   328   117   263  
Birth mother, no other adults   43   34   49   50
Birth mother & 1 male adult   24   27   20   20
Birth mother & extended family*   9   8   14   19
Biological father*   6   10   6   5
Other relative caretaker*   7   18   9   5
Other**   10   4   3   3
  N Mean N Mean N Mean N Mean
Age of respondent 306 32 324 39 116 33 260 33
Age of youngest child 311 5 328 7 117 4 263 4
Age of oldest child 311 10 328 13 117 11 263 11
Number of kids 311 3 328 3 117 3 263 3
Number of adults 311 2 328 2 117 2 263 2
* These categories may also include other non-related adults in the home.
** Includes: non-relative caretaker, adoptive or step-parent, birth mother & non-related females, or birth mother, and more than one non-related male.

At the time of referral to the Family Preservation program, families were experiencing a range of problems, some quite severe, others much less so (Table 3). Examples included one case with children ages 10 and 12 who were not enrolled in school for nearly a month and who were at risk of being removed from their home due to truancy and neglect. Another family was living in a home with no electricity, no heat, no food, no working appliances, a non-working toilet which was full of feces, and all four children slept in one bed. Yet another involved children who were sexually abused and who displayed extremely violent, uncontrollable and sexually inappropriate behavior at home and school. Although there was considerable diversity of problems, parental mental health and problematic child behavior were common issues.

 

Table 3
Selected Child and Family Problem Areas (% responding yes)
Item Kentucky
%
New Jersey
%
Tennessee
%
Philadelphia
%
Caretaker Problems
Felt blue or depressed 55 58 62 62
Felt nervous or tense 56 52 53 53
Just wanted to give up 31 33 28 33
Overwhelmed with work or family responsibility 47 56 46 52
Not enough money for food, rent, or clothing 49 52 56 56
Participation in AFDC, food stamps, WIC, social security disability, or housing vouchers 82 68 80 90
Child Problems (% of cases for which the question was relevant)
Child doesn't show much interest in what is going on 84 20 29 17
Child get(s) upset easily 69 74 60 59
Throw(s) tantrums 83 79 67 70
Fight(s) a lot with other kids 33 40 18 31
Has/Have language problems 30 26 25 18
Is/Are very aggressive toward you 43 56 18 33
Hangs with friends you don't like 28 49 44 25
Been absent from school a lot 38 42 27 19
Run away from home overnight 10 26 21 5
Been temporarily suspended from school 30 32 42 22
Been expelled from school 11 9 16 4
Took something that didn't belong to him or her 34 42 27 24
Absent from school for no good reason 30 27 18 9
Failed any classes 27 41 38 25

At the time of the first interview, approximately half of the caretakers self-reported feelings of depression or stress. In Kentucky and New Jersey, approximately half of the caretakers answered affirmatively to each of three questions about emotional difficulties: "feeling blue or depressed," "feeling nervous or tense," and "feeling overwhelmed with work or family responsibility." Caretakers in Tennessee and Philadelphia reported these difficulties at an even higher rate. Substantial proportions of caretakers reported behavioral problems in children. Between 59 and 74 percent said at least one of their children got upset easily, and two-thirds to four-fifths indicated that the children threw tantrums. Many said their children fight a lot with other kids (18% to 40%) and were very aggressive with their parents (18% to 56%). A number had problems in school, between 22 and 42 percent had children who had been suspended from school while 4 to 16 percent had children who had been expelled.

Half or more of the respondents in all four states indicated that they did not have enough money for food, rent, or clothing. About two-thirds of the respondents in New Jersey reported they participated in at least one of the five income-support programs: AFDC, food stamps, WIC, social security disability, and housing vouchers. In Kentucky and Tennessee, about 80 percent participated in one of these programs, and in Philadelphia participation was at 90 percent.

A number of families had previous involvement with the child welfare system. In Tennessee, 41 percent had previous substantiated allegations of abuse or neglect compared to 47 percent in Kentucky, 53 percent in New Jersey, and 81 percent in Philadelphia. In Kentucky and New Jersey, a fifth of the families had children who had previously been in foster care. The rate was slightly lower in Philadelphia at 17 percent. In Tennessee, only a few families had children who had previously been placed.

It might be noted that no mention is made here of substance abuse problems, thought by many to be a major issue in many families involved with the child welfare system. Very few caretakers admitted to alcohol or substance abuse in our initial interviews; fewer than five percent said they had either alcohol or drug problems. The exception was in Philadelphia and Tennessee, where 9 percent and 8 percent respectively said they "used drugs several times a week." These are likely underestimates of the extent of substance misuse in the samples particularly in Philadelphia since FPS service providers in the Philadelphia study site focused on serving families with substance abuse problems. However, other states had policies regarding referrals to family preservation that may have limited the number of families with these problems. For example, New Jersey believed that family preservation should be used cautiously for substance abuse problems. Its FPS policy manual suggested that it is unlikely that a substance abuse problem can be resolved in a 5-6 week period. In Kentucky, families in which a drug-dependent adult was not in active treatment were excluded from the program.

[Go To Contents]

Service Provision

In all sites, the caretaker interview, the caseworker interview, and the contacts data generally confirmed the expectation that the experimental group would receive more services and more intensive services than the control group (Table 4). In all four states, the number of experimental group caseworker activities reported by caretakers was greater than that reported by control group respondents, and this was also true of "helpful" caseworker activities. As for specific caseworker activities, experimental group workers in all four states were more likely to provide transportation, and talk about discipline.

 

Table 4
Summary of Services, Post-Treatment Interview
Caseworker Activities:
Proportion of affirmative answers by caretakers to yes/no questions Kentucky New Jersey Tennessee Philadelphia
  C
%
E
%
p C
%
E
%
p C
%
E
%
p C
%
E
%
p
Is caseworker still working with family 79 64 0.006 75 31 0.001 57 34 0.02      
Caseworker helped with money for rent, electricity, phone 3 17 0.001 5 4   5 10   3 4  
Caseworker helped with money for other things 9 35 0.001 10 14   11 19   5 22 .001
Caseworker provided transportation 16 42 0.001 12 25 0.003 19 34 0.10 35 50 .03
Caseworker discussed proper feeding of child 14 20   5 11 0.06 16 28   22 28  
Caseworker talked with you about discipline 35 55 0.001 39 60 0.001 46 70 0.01 32 53 .002
Caseworker talked with you on relationship with spouse 16 18   8 14 0.09 11 34 0.01 13 20  
Caseworker talked with you about how to handle anger 28 43 0.005 29 53 0.001 42 70 0.004 31 37  
Caseworker told you about other agencies 38 43   42 56 0.01 19 33 0.13 39 47  
Caseworker advised on job training programs 9 19 0.009 7 10   8 16   23 36 .04
Caseworker talked about how to get paying job 6 17 0.004 5 8   11 18   19 33 .02
Caseworker advised on how to continue school 9 18 0.04 5 8   14 23   21 34 .03
Caseworker talked about uneasy issues 27 34   29 44 0.008 22 51 0.003 27 36  
Caseworker helped you see good qualities 37 79 0.03 47 70 0.001 53 82 0.001 68 82 .01
Caseworker helped you see your problem 66 76 0.10 52 72 0.001 50 82 0.001 74 76  
Caseworker understood your situation 75 90 0.002 62 79 0.001 64 79 0.08 82 82  
NOTE: C = Control Group, E = Experimental Group

 

Table 4
Summary of Services, Post-Treatment Interview, Continued
  Kentucky New Jersey Tennessee Philadelphia
C
Mean
E
Mean
p C
Mean
E
Mean
p C
Mean
E
Mean
p C
Mean
E
Mean
p
CT report of # of caseworker activities 2.18 3.90 0.0001 2.31 3.25 0.001 2.89 4.60 0.02 2.9 4.6 .0001
CT report of # of "helpful" caseworker activities 1.04 1.68 0.0001 1.11 1.97 0.0001 0.83 1.33 0.04 1.5 2.2 .02

 

Services Provided:
Proportion of affirmative answers by caretakers to yes/no questions
  Kentucky New Jersey Tennessee Philadelphia
C
%
E
%
p C
%
E
%
p C
%
E
%
p C
%
E
%
p
Anyone been in job training program 3 8 0.09 2 3   3 4   20 26  
Anyone been in WIC 32 45 0.02 22 20   51 41   40 44  
Been in a marriage counseling program 0 7 0.006 2 2   0 1   2 2  
Anyone receive daycare 5 19 0.001 10 7   26 26   11 15  
Anyone receive transportation 7 16 0.02 14 12   17 19   25 39 .02
Anyone receiving parent education/training 13 19   6 10   20 8 0.06 16 37  
Anyone receive counseling 35 52 0.003 50 56   >9 17   21 26  
Anyone receive help finding a place to live 1 4   5 2   17 5 0.04 9 9  
Anyone stay at an emergency shelter 1 1   2 1   6 0 0.03 4 3  
Anyone receive medical or dental care 8 15 0.07 36 42   34 16 0.03 33 39  
Anyone receive homemaker services 1 3   6 3   14 3 0.02 1 1  
Were any needed services not gotten 27 19   56 42 0.01 39 24 0.10 24 19  
  C
Mean
E
Mean
p C
Mean
E
Mean
p C
Mean
E
Mean
p C
Mean
E
Mean
p

Caseworker report of # of services provided

3.16 4.99 0.001 2.31 3.17 0.001 1.58 3.19 0.0002 3.4 4.9 .0004
NOTE: C = Control Group, E = Experimental Group
Table only includes items with a primary p-value less than .05 in at least one of the states; p-values greater than .20 are not reported.
Items in bold indicate significant findings in favor of the experimental group; italicized items indicate significant findings in favor of the control group.

Central casework activities with families included counseling families, handling anger, and child discipline. These activities reflect common problems with families that are of paramount concern to the child protective system. Experimental group caseworkers in the Homebuilders states were more often reported to have talked about difficult issues, to have helped the caretaker to see her/his good qualities and problems, and to have understood the parent's situation. In Philadelphia, caretakers reported much the same.

Insofar as there are differences between groups, it can be assumed that the experimental conditions held since the experimental group received substantially more services than the control group. As is to be expected in real life implementations of models, the programs did not adhere completely to the Homebuilders approach as described above. In addition to other critical elements of family preservation, the Homebuilders model specifies that workers should provide an in-home contact within 72 hours of referral, and family preservation workers should be available 7 days per week. Substantial contact should take place within the first week; the model's developers suggest that the typical case receive 11 hours of service in that time. Concrete services are also an important component of service, particularly early in the case. Based on caseworker reports, families did not always receive contact within 72 hours, fewer than expected contacts occurred in the first week of the program, and few contacts occurred on weekends. There was relatively little provision of concrete services early on.

[Go To Contents]

Findings

This evaluation of family preservation programs was designed to assess the extent to which key goals of the programs are being met: the goals of reducing foster care placement, maintaining the safety of children, and improving family functioning. The assessment of effects on placement and safety of children was based on administrative data, which were available on families for at least one year after the beginning of service. Family functioning was assessed through interviews with caretakers at the beginning of service, one month later (at the end of service for the family preservation group), and a year after the beginning of service. Interviews with caseworkers were also conducted at the beginning and one month points.

No significant differences were found between the experimental and control groups on family level rates of placement, case closings, or subsequent maltreatment. There were a few child and family functioning items in which the experimental group displayed better outcomes than the control group in at least one of the states. However, these results did not occur in more than one state. It was found that family preservation programs in two states resulted in higher assessments by clients of the extent to which goals have been accomplished and of overall improvement in their families' lives.

Reducing Foster Care Placement. In none of the four states were there statistically significant differences between the experimental and control groups on family level rates of placement or case closings (Table 5). In Kentucky, placement rates at the end of one year were 25 and 24 percent for the experimental and control groups, respectively. In New Jersey, the percents were 29 and 22 percent. The rates in Tennessee were 23 and 19 percent. In Philadelphia, placement rates were 18 and 15 percent at the end of one year.

Table 5
Summary of Placement Data, Survival Analyses Percents of Families Experiencing Placement of
at Least One Child Within Specified Periods of Time
Kentucky 1 month 6 months 12 months 18 months
E C E C E C E C
Primary analyses 6 5 18 18 25 24 27 27
Secondary analyses 4 4 12 18 20 23 24 25
Refined analyses
Investigative 8 5 15 14 26 15 28 20
Recent substantiation 6 2 20 11 29 13 32 18
Petition cases 6 9 16 14 22 29 25 32

 

New Jersey 1 month 6 months 12 months 18 months
E C E C E C E C
Primary analyses 5 6 19 17 29 22 35 26
Secondary analyses 3 6 17 17 27 23 34 27
Refined analyses
Investigative 3 5 16 12 25 15 32 19
Recent substantiation 8 5 19 12 25 14 33 21

 

Tennessee 1 month 6 months 12 months
E C E C E C
Administrative data, primary analysis 11 11 22 19 23 19
Administrative data, secondary analysis 7 12 18 19 19 19
Including relatives, primary 11 11 26 21 28 23
Including relatives, secondary 7 12 20 19 23 21
Refined analyses
Recent investigation, CORS* 7 12 15 15 17 15
Recent investigation, includes Relative 7 12 18 18 22 21

 

Philadelphia 1 month 6 months 12 months 18 months
E C E C E C E C
Primary analyses 1 1 10 12 18 15 24 20
Secondary analyses 1 1 9 13 15 16 21 19
* Client Operation and Review System

As to be expected with any program, some of the families assigned to family preservation programs did not receive the services or received a minimal dosage of the services. In addition, a small number of the families in the control group were actually provided family preservation services. To address these issues, analyses were conducted in which these cases were dropped (secondary analysis). Results of the secondary analyses were quite similar to the primary analyses, also showing no significant differences between the groups in rates of placement.(5)

The ideal family preservation case is one in which there has been a recent significant crisis in the family, resulting in the maltreatment that triggers the possibility of removal of the child from the home. Subsamples of cases that approached this ideal were examined. Again in these analyses, there were no statistically significant differences between the experimental and control groups in placement rates over time.

In addition to placement rates at various points in time, placement was examined in terms of proportion of time in substitute care after random assignment. No significant differences were found in care days for the families in any of the four states. In Kentucky, both the experimental and control group children spent an average of 6 percent of the days after random assignment in care. In New Jersey and Philadelphia, experimental group children spent an average of 6 percent of that time in placement compared to 4 percent for the control group children. In Tennessee, experimental group children spent an average of 10 percent of that time in placement, compared to 5 percent for the control group children.

Targeting. Since these programs were intended to prevent the placement of children, the target group for the Homebuilders program services was families in which at least one child was "in imminent risk of placement."(6)

As in previous studies, it was found that most of the families served were not in that target group. This is shown by the placement rate within a short period in the control group, indicating the placement experience in the absence of family preservation services. In all three states, the placement rate in the control group within one month was quite low. It would, therefore, have been virtually impossible for the programs to be effective in preventing imminent placement, since very few families would have experienced placement within a month without family preservation services.

A number of subgroups that were thought to represent better targeting were examined. These included:

  • cases coming directly from the investigation of an allegation of abuse or neglect,
  • cases with recent substantiated allegations of abuse or neglect,
  • cases in a Kentucky subgroup in which workers had submitted petitions to the court for placement or some other court-ordered intervention.

In none of these subgroups did placement rates in the control group within one month exceed 12 percent. Hence, even in these more refined (from the standpoint of targeting) subgroups the intended target group, children in imminent risk of placement, was not in evidence.

It should be noted that the results found here occurred despite efforts in this project to improve targeting. In Kentucky and New Jersey, a special screening form, developed by the evaluation team, was employed to rate the risk to children with the intent that cases with intermediate risk would be referred to the program. In Kentucky, efforts were made to divert to family preservation cases that had been referred to the court. In Tennessee, special training efforts were instituted to address concerns about targeting.

Child Safety. Maltreatment after the beginning of service was generally not related to experimental group membership, except for one subgroup in Tennessee. Subsequent maltreatment was measured by the occurrence or nonoccurrence of a substantiated allegation of abuse or neglect following an investigation of such an allegation. The rate of subsequent maltreatment was relatively low, about 18 percent of the families in Kentucky had a substantiated allegation within one year of random assignment; in New Jersey the rate was 12 percent and in Tennessee, 25 percent. In Tennessee, in those families with an allegation within 30 days prior to random assignment, the experimental group children experienced fewer substantiated allegations than children in the control group did.

The findings of little difference between the experimental and control groups in subsequent maltreatment can be read in two ways. It indicates that families served by family preservation were no more likely than families not receiving the services to be subjects of allegations of harm. In this sense, children were, largely, kept safely at home while receiving family preservation services. However, children in both groups were primarily in their homes, and family preservation did not result in lower incidence of maltreatment compared with children in the control group.

Subgroups. In an effort to identify groups of cases for which family preservation is effective, subgroups of Kentucky, New Jersey, and Philadelphia cases were examined.(7) Subgroups were defined in terms of problems of the family (e.g., substance abuse, financial difficulties, and depression) and family structure. Within these subgroups, experimental and control groups were compared on placement and substantiated allegations after random assignment. Two significant differences were found. Among single mothers in New Jersey, those in the experimental group were less likely to have a subsequent substantiated allegation than those in the control group. Among families in Philadelphia who identified a child having problems with school, those in the experimental group were more likely to have a substantial allegation than those in the control group. No subgroups were found in which there were effects on placement in any state.

Family Functioning. In a few areas of family functioning, across states, families in the experimental group appeared to be doing better at the end of services. There were very few differences at the year follow-up and in changes over time. Those differences that did appear (primarily at the end of services) were not consistent across states and were not maintained. Family functioning was assessed through caretaker and caregiver interviews at three points in time  shortly after the beginning of services, four to six week later (at the end of services for the Homebuilders group), and again a year after services began. Differences between groups at post treatment, follow-up, and change over time are presented in Table 6.

 

Table 6
Summary of Family and Child Functioning Outcomes
Differences Between Experimental and Control Groups
at Post Treatment, Followup, and Change Over Time
Area Post treatment Follow-up
(1 year after start of treatment)
Change over time
Life events
Positive life events KY: Ø
NJ: Ø
TN: Ø
PA: Fewer experimentals experienced positive life events
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Negative life events KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Depression KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Family problems, individual items KY: Ø
NJ: fewer experimentals not enough money for food, rent, or clothing
TN: fewer experimentals had few or no friends PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
N/A
Economic functioning
Individual items KY: Ø
NJ: fewer experimentals difficulty paying rent and buying clothes
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: fewer experimentals having difficulty paying rent
PA: more experimentals having difficulty buying food and clothes
N/A
Scale KY: Ø
NJ: experimental average lower (better)
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Household condition
Individual items KY: experimentals had fewer broken windows or doors
NJ: Ø
TN: more experimentals in unsafe building because of illegal acts
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: more experimentals reporting not enough basic necessities
N/A
Scale KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Experimental group reporting more problems in household condition
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Child care practices
Individual items KY: fewer experimentals used punishment for not finishing food
NJ: experimentals less often got out of control when punishing child and more often encouraged child to read a book
TN: more experimentals went to amusement park, pool, or picnic
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
 
Positive scale KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Negative scale KY: Ø
NJ: experimentals lower (better)
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Punishment KY: Ø
NJ: experimentals lower (better)
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Caretaker depression KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Child behavior
Aggression KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
School problems KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Positive child behaviors KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Negative child behaviors KY: Ø
NJ: experimental group lower (better)
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Overall assessment of improvement KY: experimentals, greater improvement
NJ: experimentals, greater improvement
TN: Ø
PA: Ø
KY: Ø
NJ: Ø
TN: Ø
PA: Ø
N/A
Caseworker report of caretaker functioning
Individual items KY: Ø
NJ: control group higher (better) in ability in giving affection and providing learning opportunities
TN: experimental group higher (better) on five items
PA: Ø
N/A KY: control group had more positive change in respecting child's opinions
NJ: control group had more positive change in respecting child's opinions
TN: experimental group more positive change on setting firm and consistent limits
PA: Ø
Scale KY: Ø
NJ: Ø
TN: experimental group higher (better)
PA: Ø
N/A KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Caseworker report of household condition KY: control group better
NJ: control group better
TN: Ø
PA: control group worse
N/A KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Caseworker report of caretaker problems KY: experimentals more problems
NJ: Ø
TN: Ø
PA: Ø
N/A KY: Ø
NJ: Ø
TN: experimentals declined more PA: Ø
Caseworker report of child problems KY: Ø
NJ: Ø
TN: Ø
PA: Ø
N/A KY: Ø
NJ: Ø
TN: Ø
PA: Ø
Ø denotes that differences between groups were not significant at p <=.05; N/A denotes not applicable.

Areas assessed included life events, economic functioning, household condition, child care practices, caretaker depression, child behavior, and caretaker functioning. It can be said that family preservation services may have small, apparently short-term, effects on some areas of functioning. There was one item with some consistency across sites, the overall assessment of improvement by caretakers. At post treatment, a significantly larger proportion of experimental group caretakers in Kentucky and New Jersey generally thought there was "great improvement" in their lives. In Tennessee and Philadelphia, although not significant, results tended in the same direction.

Implications

The findings of this study are not new. A number of previous evaluations with relatively rigorous designs have failed to produce evidence that family preservation programs with varying approaches to service have placement prevention effects or have more than minimal benefits in improved family or child functioning. The work reported here may be thought of as four independent evaluations in four states, adding to the set of previous studies with similar results, this time focusing on Homebuilders programs. The accumulation of the findings from a number of studies in several states, with varying measures of outcome, is compelling.

The findings should not be taken as showing that these programs serve no useful purpose in the child welfare system. The results can be seen as a challenge to keep trying, to find new ways to deal with the problems of families in the child welfare system. The findings indicate the grave difficulties facing those who devise approaches to these problems. Failure in such undertakings should not be surprising and those who risk trying to find solutions should not be punished when evaluations such as this indicate they may have come up short.

The accumulation of findings suggests that the functions, target group, and characteristics of services in programs such as this need to be rethought. Obviously, function, target group, and services are closely intertwined. The foremost of these issues concerns the objectives of the programs. A number of observers have suggested that placement prevention be abandoned as the central objective in intensive, family preservation services in favor of other objectives, notably the improvement of family and child functioning. Targeting these services on families at risk of placement is unlikely to be successful. So if these services are to continue, they will continue to serve "in-home" cases and families in which there has been a substantiated allegation of abuse or neglect or serious conflicts between parents and children where children remain in the home. Many, if not most, of these "intact" families need help. Relatively intensive and relatively short-term services such as those provided by family preservation programs are one source of such help. In this respect, family preservation programs can be thought of as an important part of the continuum of child welfare services.

Another question that program designers must address is that of specialization. Subgroups for which the program was successful were not found, but these programs are quite general in character, and thus may sacrifice some of the benefits of specialization. Those benefits are a clearer focus of services, a tighter target group definition, specification of service characteristics (such as length and intensity based on needs of the target group), and the development of more specific competencies on the part of workers. Specialization could be in terms of problems (e.g., substance abuse) or characteristics of clients (young, isolated mothers). There are clear drawbacks to specialization, including the tendency to define problems in terms of the service one offers. Furthermore, limiting target groups inherently limits the impact of programs. Nonetheless, it may be better to mount a series of small programs rather than putting all of one's resources into large, undifferentiated efforts.

Program planners must also address the issue of length and intensity. The extent to which the intensive, short-term, crisis approach fits the needs of child welfare clients should be reexamined. The lives of these families are often full of difficulties--externally imposed and internally generated--such that their problems are better characterized as chronic, rather than crisis. Short-term, intensive services may be useful for families with chronic difficulties, but those services are unlikely to solve, or make much of a dent in the underlying problems. Of course, the hope is family preservation programs will be able to connect families with on-going services to treat more chronic problems. But, that appears to happen far less than needed. The central point here is that we need a range of service lengths and service intensities to meet the needs of child welfare clients. It is essential that policy makers, planners, and program providers maintain realistic expectations of the effects of short-term family preservation programs.

[Go To Contents]

Endnotes

1.  This is one of two reports completed for the evaluation. A previous report, The Evaluation of the New York City HomeRebuilders Demonstration reported on a program designed to facilitate the reunification of children in foster care with their families.

2.  As to be expected with any program, some of the families assigned to family preservation programs did not receive the services or received a minimal dosage of the services. In addition, a small number of the families in the control group were actually provided family preservation services. To address these issues, analyses were conducted in which these cases were dropped (secondary analysis).

3.  J. Littell and J. Schuerman. (1995). A Synthesis of Research on Family Preservation and Family Reunification. http://aspe.hhs.gov/hsp/cyp/fplitrev.htm.

4.  Jill Kinney, David Haapala, and Charlotte Booth. (1991). Keeping Families Together: The Homebuilders Model. New York: Aldine de Gruyter.

5.  It should be noted that the most rigorous approach to analysis requires that cases be maintained in the groups to which they were randomly assigned. Random assignment is used to assure that the groups are as similar as possible at the outset of service. Removing cases from the groups or switching cases from one group to another threatens group equality and allows for the possibility that post-treatment differences could be explained by factors other than service. In particular, it is likely that violations and minimal service cases differ in systematic ways from other cases. Hence, the secondary analyses should be viewed with caution.

6.  The Philadelphia FPS program did not target imminent-risk children.

7.  The number of cases in Tennessee was too small to allow subgroup analysis.

Preview
Download

"execsum.pdf" (pdf, 190.85Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®