Evaluation of Family Preservation and Reunification Programs: Final Report - Volume Two. 4.5.5 Summary of Outcome Data

09/01/2002

Information from the caretaker interviews, the caseworker interviews, and the administrative data were analyzed for indications of differences between the experimental and control groups subsequent to the referral to the family preservation program. Tables 4-28 and 4-29 contain a summary of those outcomes on which we found significant differences between the experimental and control groups in the primary analysis (p < .05). Items in bold are those on which the experimental group had better outcomes, those in italics are those on which the control group had better outcomes.

There were no significant differences between experimental and control groups on family level rates of placement. Subsequent maltreatment was generally not related to experimental group membership either.

Table 4-28
Summary of Outcomes in Philadelphia, Post-treatment Interview
Caretaker Scales: Control
Mean
Experimental
Mean
p
Positive life events .15 .19 .05
Caseworker Scales:
Household Condition
(higher = worse condition)
.16 .13 .05
NOTE: This table only includes items with a primary p-value less than or equal to .05. Items in bold indicate significant findings in favor of the experimental group.
Table 4-29
Summary of Outcomes in Philadelphia, Caretaker Followup Interview
Proportion of affirmative answers to yes/no questions
  Control
%
Experimental
%
p
Not enough money for food, rent, or clothing 33 49 .02
Had difficulty buying enough food 16 27 .05
Had difficulty buying clothes 21 37 .01
Punished children for not finishing food 1 7 .05 (FE)
NOTE: This table only includes items with a primary p-value less than or equal to .05.
Italicized items indicate significant findings in favor of the control group.
"FE" indicates significance determined by Fisher's exact test.
Caretaker Scales: Control
Mean
Experimental
Mean
p
Stolen things or arrested .13 .26 .02
Number of 10 household condition problems .06 .10 .05
NOTE: This table only includes items with a primary p-value less than or equal to .05.
Italicized items indicate significant findings in favor of the control group.

As shown in Tables 4-28 and 4-29, there were few significant differences between experimental and control groups in analyses of child and family functioning items. It should also be noted that the results have not been adjusted for the multiplicity of significance tests performed. That is, these significant items surfaced out of a large number of items and scales examined. In such a situation it is to be expected that some items will show significant differences simply by chance, so the appearance of a few significant differences should not be taken as an indication of superiority of one group over another. Overall, we are unable to claim consistent evidence of positive effects of the family preservation services in Philadelphia that were examined in this study.

Endnotes

57. There was a small, non-significant difference between the experimental group, of whom 5.8 percent said yes, and the control group, of whom 11.2 percent responded yes (p = .11).

58. Although not significantly different, compared to the control group, the experimental group did have a somewhat higher proportion of respondents who reported having been neglected as a child (27% vs. 19%; p = .15).

59. The state of Philadelphia reports 8 possible outcomes for reports of maltreatment; (1) indicated, perpetrator admitted, (2) indicated, medical evidence, (3) not substantiated, (4) pending determination, (5) substantiated, (6) unfounded, (7) indicated - investigating, and (8) unable to complete.

60. Placement spells are defined as any consecutive period of time in substitute care and may consist of several distinct placements (i.e., several different foster homes).

61. The results of "secondary" analyses, dropping violations of random assignment and cases receiving minimal service show slightly greater differences between the experimental and control groups (see Appendix). Here, the same 8 significant primary analysis items show significant differences in favor of the experimental group at p = .01 or lower, and one additional item showed significant differences in the same direction at p < .05. All nine items that showed significant differences in the secondary analysis remained significantly different in the tertiary analysis (see Appendix).

62. These differences remained significant and in the same direction for both secondary and tertiary analyses (for magnitude and significance levels, see Appendix).

63. Differences remained significant and in the same direction for both secondary and tertiary analyses (for magnitude and significance levels, see Appendix).

64. In the secondary analysis (dropping violations and minimal service cases) and the tertiary analysis (dropping additional cases that may not have had a worker assigned), a significantly greater proportion of experimental group caretakers reported receiving transportation and parent education/training classes (see Appendix for magnitude and significant levels).

65. The difference was, however, significant in the secondary analysis (21% control vs. 14% experimental, p = .001) and the tertiary analysis (22% control vs. 13% experimental, p = .003).

66. In addition to slight changes in the magnitude of the difference in whether workers helped caretakers see their good qualities, secondary analysis revealed that a significantly greater proportion of experimental group caretakers reported that they and their worker agreed on goals most of the time (75% vs. 70%, p = .03). Tertiary analysis revealed no additional items with significant differences (see Appendix).

67. When violations and minimal services cases were excluded, the difference between the groups was even larger (5.3 vs. 3.1, p = .0001).

68. SSI for adult or child was reportedly provided more often to the control group than the experimental group, and the difference was marginally significant by Fisher's Exact test (3% vs. 0%; p = .06). In the secondary analyses, excluding violations and minimal service cases, 10 services were provided significantly more often to the experimental group than to the control group (again, significance levels were all at p = .05 or less). In addition to the 6 primary analysis items showing differences in favor of the experimental group, secondary analyses indicate that the following services were also provided significantly more often by the experimental group: health assessment, housing financial assistance, self help groups, and homemaker services (see Appendix for magnitude of difference and significance levels). In the secondary analyses no services were provided significantly more often to the control group than the experimental group.

69. None of the items showed significant differences between experimental and control group caretakers in the secondary analysis or the tertiary analysis.

70. Secondary and tertiary analyses did not result in any significant differences either.

71. This difference remained significant in the secondary analysis (31% vs. 16%; p = .02) and was marginally significant in the tertiary analysis (31% vs. 18%; p = .06).

72. Tertiary analyses were not performed on caseworker interview data due to the fact that all of the 29 additional cases dropped for this level of analysis were missing both caseworker interviews and results would therefore be the same as for the secondary analysis.

73. In determining placements, we depended on the variable "factype" in the administrative data. The specific categories for this variable included: adoption, foster care, private institution/boarding schools, family treatment home, unmarried parent, other, children's psychiatric hospital, and foster care medically fragile.

74. Due to the "clustering effect," analyses at the child level are misleading. Clustering refers to the lack of independence between children within the same family of observations of such things as placement. One could argue that if one child is removed from the home, the remaining children are more likely to experience placement. The "clustering effect" leads to an underestimate of the significance levels when analyses are conducted at the child level. Conducting the analyses at the family level is one approach to resolving this dilemma.

75. Fifteen percent of the experimental group, and 16 percent of the control group experienced substitute care placement within a year in the secondary analysis.

76. Analyses were also done on all allegations, whether substantiated or not. The results were very similar, although, of course, rates for all allegations were higher.

77. Figure 4-2 also displays the survival curves for the secondary analysis group. At one year, 22 percent of the experimental group and 13 percent of the control group experienced substantiated reports of maltreatment.

78. This difference remained significant in both the secondary and tertiary analyses (see Appendix for magnitude of difference and significant levels).

79. These results held for the secondary and tertiary analyses.

80. These results were maintained in the secondary and tertiary analyses.

81. This difference was slightly larger and more significant in the secondary analysis (53% experimental and 33% control; p = .006) and the tertiary analysis (53% experimental and 31% control; p = .005). Tertiary analysis also revealed significant differences in the proportion of respondents indicating they "just wanted to give up," with a greater proportion of experimental group respondents answering affirmatively (29% vs. 15%; p = .04).

82. There were no significant differences between experimental and control groups on these items in the secondary or tertiary analyses.

83. The difference was greater and marginally significant in the secondary analysis (.30 vs. .21; p = .06). In the tertiary analysis, the difference was greater still and it was statistically significant (.31 for the experimental group and .19 for the control group; p = .02).

84. Differences on both items remained significant in the secondary and tertiary analyses (see Appendix for magnitude of differences and significance levels).

85. This difference increased and remained significant in the secondary analysis (25% vs. 11%; p = .009) and the tertiary analysis (26% vs. 11%; p = .01). In the tertiary analysis, one additional item resulted in significant differences at the time of the post-treatment interview. Six percent of the experimental group caretakers and none of the control group caretakers reported that "there were bare electric wires" (Fisher's exact p-value = .02).

86. This difference was not significant in either the secondary or the tertiary analysis. However, tertiary analyses revealed that two different items resulted in significant differences between experimental and control group caretakers at the time of the follow-up interview. Nine percent of the experimental group caretakers and none of the control group caretakers reported that the "plumbing did not work" (Fisher's exact p-value = .01). Seventeen percent of the experimental group caretakers and six percent of the control group caretakers reported that "a lot of paint was peeling" (p = .04).

87. In the secondary analysis, "hitting child harder than meant to" was the only item for which there were significant differences between the experimental and control groups, with a greater proportion of the experimental group responding affirmatively (13% vs. 5%, p = .03). The difference was not significant in the tertiary analysis.

88. This difference was not significant in the secondary or tertiary analysis.

89. Reliability analysis yielded a Cronbach's alpha of .92 at initial interview, .90 at post-treatment, and .94 at follow-up.

90. These results held for the secondary and tertiary analyses.

91. This difference was greater and statistically significant in the secondary analysis (31% vs. 18%; p = .02) and the tertiary analysis (31% vs. 16%; p = .02).

92. These results were also maintained in the secondary analysis (80% vs. 63%; p = .02) and tertiary analysis (79% vs. 62%; p = .03).

93. Secondary analyses resulted in significant differences on three of the individual items. Relative to caseworkers in the control group, caseworkers in the experimental group rated caretakers higher (more adequate) on "respecting children's opinions" (2.84 vs. 2.56; p = .05), "setting firm limits for children" (2.50 vs. 2.18; p = .05), and "providing adequate personal supervision" (2.86 vs. 2.51; p = .04).

94. These results were maintained in the secondary analysis (.13 vs. .17; p = .03).

95. This was also true in the secondary analysis.

96. This result held in the secondary analysis.

View full report

Preview
Download

"report2.pdf" (pdf, 978.24Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®