Evaluation of Family Preservation and Reunification Programs: Interim Report. 8.10 Summary of Outcome Data

01/08/2001

Information from the caretaker interviews, the caseworker interviews, and the administrative data were analyzed for indications of differences between the experimental and control groups subsequent to the referral to the family preservation program. Tables 8-15 and 8-16 contain a summary of those outcomes on which we found significant differences between the experimental and control groups in any state for the primary analyses (p < .05). Items in bold are those on which the experimental group had better outcomes, those in italics are those on which the control group had better outcomes.

In none of the three states were there significant differences between the experimental and control groups on family level rates of placement or case closings. Subsequent maltreatment was generally not related to experimental group membership, except for one subgroup in Tennessee. In Tennessee, in those families with an allegation within 30 days prior to random assignment, the experimental group children experienced fewer substantiated allegations than children in the control group.

In Tables 8-15 and 8-16 there are a number of child and family functioning items in which the experimental group displayed better outcomes than the control group in one of the states. It should be noted that the results have not been adjusted for the multiplicity of significance tests performed. That is, these significant items surfaced out of a large number of items and scales examined. In such a situation it is to be expected that some items will show significant differences simply by chance, so the appearance of a few significant differences should not be taken as an indication of superiority of one group over another, particularly when the results are not confirmed in more than one state. On only two items were differences found in two states: caretakers' assessment of whether goals had been accomplished and their assessment of overall change. We are inclined to believe that family preservation programs as represented in these states do result in higher assessments by clients of the extent to which goals have been accomplished and of overall change, since differences on those items were found in both states. Beyond that, we are unable to claim consistent evidence of positive effects of family preservation services.32

There are a few items on which the control group had better outcomes, nearly all of them on measures provided by caseworkers. We are not inclined to read too much into these results, since experimental group caseworkers generally knew the families better and there may well have been significant differences in the ways that workers serving the two groups saw families and judged their functioning.

 

Table 8-15
Summary of outcomes, post-treatment interview
Caretaker Interview: Proportion of affirmative answers to yes/no questions  Kentucky New  Jersey Tennessee
Control % Exp % p Control % Exp % p Control % Exp % p
Is apartment/house rented (vs. owned) 75 89 0.005 70 68   69 75  
Got together with anyone to have fun 64 64   65 59   38 75 0.001
Felt had few or no friends 14 18   20 18   38 19 0.03
Had difficulty buying clothes 17 21   47 33 0.008 27 24  
Out of control when punishing child 24 24   40 30 0.05 11 12  
Punished for not finishing food 7 1 0.02 6 5   0 0  
Unable to find someone to watch child 9 12   21 12 0.04 20 27  
Encouraged child to read a book 92 90   82 91 0.02 94 96  
Have goals been accomplished 63 77 0.02 52 71 0.001 81 84  
Assessment of overall change:     0.02     0.001      
Some Improvement 16 22   9 16   32 32  
Great Improvement 31 42   41 52   32 42  
Same 42 29   34     22 14  
Same Somewhat or a great deal worse 12 6   16 12   14 13  
Caretaker Scales:
Difficulty paying bills (proportion of 4 items) 0.17 0.22   0.34 0.25 0.02 0.25 0.18  
Negative child care practices (proportion of 10 items) 0.14 .0.13   0.18 0.14 0.02 0.09 0.09  
Punishment (proportion of 5 items) 0.16 0.17   0.25 0.20 0.04 0.13 0.13  
Negative child behaviors (proportion of 21 items) 0.34 0.34   0.33 0.28 0.04 0.21 0.21  
Change in proportion of punishment items from Initial to Post-treatment interviews -0.04 -0.09 0.05 -0.05 -0.07   -0.07 -0.13  
Change in proportion of negative child care practices from Initial to Post-treatment interviews -0.02 -0.06 0.04 -0.04 -0.05   -0.01 -0.08 0.02
Caseworker Scales:
Ability giving affection (higher = more adequate) 2.83 2.83   2.93 2.70 0.04 2.73 2.95  
Providing learning opportunities for child (higher = more adequate) 2.38 2.42   2.89 2.60 0.008 2.64 2.64  
Respecting child's opinions (higher = more adequate) 2.58 2.45   2.55 2.42   2.35 2.84 0.01
Responding patiently to child's questions (higher = more adequate) 2.43 2.34   2.44 2.27   2.26 2.67 0.04
Adequate supervision / Responsible child care (higher = more adequate) 2.50 2.59   2.80 2.71   2.52 2.93 0.04
Household condition (proportion of 13 items, higher = worse condition) 0.10 0.13 0.01 0.09 0.11 0.02 0.12 0.12  
Caretaker problems (proportion of 21 items, higher = more problems) 0.25 0.31 0.0005 0.21 0.23   0.21 0.18  
Caretaker functioning (higher = better) 2.56 2.55   2.79 2.66 0.10 2.51 2.82 0.04
Respecting child's opinions (change in average ratings from Time 1 to Time 2)** 0.19 -0.06 0.05 0.27 0.04 0.05 0.06 0.14  
Setting firm/consistent limits/rules (change in average ratings from Time 1 to Time 2) ** 0.35 0.22   0.33 0.25   -0.29 0.29 0.01
Caretaker Problems (Change in proportion of 21 items; lower = less at Time 2) -0.06 -0.04   -0.05 -0.04   -0.01 -0.08 0.05

NOTE: This table only includes items with a primary analysis p-value less than .05 in at least one of the states; p-values greater than .10 are not reported.

Items in bold indicate significant findings in favor of the experimental group whereas italicized items indicate significant findings in favor of the control group.

** Scale for change in ratings: -4 = ability decreased greatly over time, 0 = no change in ability over time, +4 = ability increased greatly over time


Table 8-16.
Summary of outcomes, caretaker follow-up interview
Proportion of affirmative answers to yes/no questions
  Kentucky New Jersey Tennessee
Control % Exp % p Control % Exp % p Control % Exp % p
Has spouse held full time job 81 78   86 68 .05 100 85  
Had difficulty paying rent 20 20   34 27   39 20 .04
Have children handled household chores 75 75   70 83 .02 94 89  

NOTE: This table only includes items with either a primary p-value less than .05 in at least one of the states; p-values greater than .10 are not reported

Items in bold indicate significant findings in favor of the experimental group whereas italicized items indicate significant findings in favor of the control group.


(32)  The reader is reminded of the findings reported in Chapter 7 indicating that experimental group caretakers generally had more positive views of service and of their relationships with workers than control group caretakers.