Evaluation of the New York City Home Rebuilders Demonstration. 3.2.3 Outcomes

09/14/1998

The interviews included a number of questions that measure outcomes. Some of these items had very little variation in response. The relationship between membership in experimental and comparison groups was examined for those items that had at least minimal variation. In addition, summations of various individual items formed scales. Differences between the experimental and comparison groups were analyzed within the four agencies in which we interviewed caretakers in both groups (HD, LF, MM, and OTT). Three of these agencies (HD, LF, and MM) were considered "true experimental" agencies. We did not analyze combinations of all experimental groups against all comparison groups, since such combinations would have little meaning, given the lack of control groups in JC and NYF and the nonrandom formation of groups in OTT.

Three categories of significant findings were identified, those at or below p = .1, .05, and .01. Probabilities reported are two-tail, based on Chi-square, Fisher exact, or t-tests, as appropriate. T-tests of differences in group means were used for scales and for a few items involving ordered categorical responses. A large number of statistical tests were performed and this invariably leads to the identification of spurious results, so caution must be used in interpreting isolated significant findings.

Parent and Family Functioning. Parent and family functioning was examined only for respondents who were birth parents. Although the functioning of other caretakers and their families is not unimportant, HomeRebuilders was originally conceived as a program that would enable birth parents to regain custody of their children. The focus of the intervention was to be on the functioning of those parents. Furthermore, when other goals were pursued (most notably, adoption) involving other caretakers, the extent of involvement of the agencies with those other caretakers varied considerably, making comparisons problematic. Unfortunately, the numbers of birth parents within experimental and comparison groups within agencies was very small, ranging in these analyses from 6 to 19, making it difficult to detect significant differences. Table 3-16 summarizes the results.

Table 3-16.  Birth parent and family functioning

Items and Scales Number of Items Number of E-C Comparisons for 4 Agencies Results of Within Agency Comparisons of Experimental and Comparison Groups
Personal life events (Q41) 14 (7 analyzed) 28 No significant differences
Depression subscale 4 4 No significant differences
Caretaking behaviors (Q55) 15 (8 analyzed) 32 No significant differences
Punishment subscale 5 4 LF exp group higher levels of punishment, p < .05
Positive family life events scale (Q40) 6 4 No significant differences
Negative family life events scale (Q40) 8 4 No significant differences
Difficulties paying for things scale (Q52) 4 4 OTT exp group less difficulty, p < .1
Overall assessment of changes in family life (Q69) 1 4 No significant differences

There were two series of yes-no questions. The first (Personal Life Events) asked whether any of 14 "things have happened to you in the last three months," and included both positive ("gotten together to have fun or relax," "felt happy") and negative ("felt blue or depressed," "felt nervous or tense," "felt you had few friends," "not enough money for essentials," "overwhelmed by work or family responsibilities") experiences (caretaker interview, question 41). Seven of these items had too little variation to analyze further. On none of the remaining seven items were there any significant (p < .1) differences between comparison groups in any agency. Sometimes the experimental group had better outcomes, other times the comparison group. Four of the items could be thought of as reflecting depression and were combined into a "life events depression" scale. None of the comparisons of the birth parents in the experimental and comparison groups were significant on this scale.

The second series of yes-no questions was 15 items tapping caretaking behaviors ("In the last three months, have you . . .") (caretaker interview, question 55). Seven of these items had too little variation to compare experimental and comparison groups. Of the remaining eight items, five were positive ("praised your child for doing well," "listened to music together," "gone to an amusement park, pool, etc.," "encouraged your child to read a book," and "had your child handle household chores on a regular basis") and three were negative ("lost your temper when your child got on your nerves," "things got out of control when punishing your child," and "blamed your child when things were not his or her fault"). Five of the fifteen items formed a "punishment" scale.

On the eight items with adequate variation, none of the 32 within agency comparisons of birth parent respondents proved significant. For the punishment scale derived from these items, there was one significant difference: the experimental group in LF had a higher mean score (higher level of punishment) than the control group (p < .05).

The interview included a family life events inventory of 15 items ("has anything like this happened to you or someone in your household in the past three months") (caretaker interview, question 40) from which were derived two scales, positive life events (6 items) and negative life events (8 items). None of the within agency experimental-comparison group comparisons for birth parents were significant on these scales. Four yes-no questions were asked about difficulties in paying for things, from which a scale was derived consisting of a count of yes responses (caretaker interview, question 52). Only in OTT was there a significant difference on this scale: the experimental group had fewer problems in paying for things than the comparison group (p < .1).

Respondents were asked for their overall assessment of changes in family life since the agency began work with them (caretaker interview, question 69). A five point scale, ranging from great improvement (1) to a great deal worse (5) was provided. Examining means on this item, there were no significant differences between experimental and comparison groups within agencies, for either all respondents or for birth parents only. Means were lower (better) for the experimental group in HD and LF and the comparison group in OTT. In MM the means were the same for all respondents and lower in the control group for birth parents. Responses to this question, collapsed into three categories, were also examined in crosstabulations, (the categories "same," "somewhat worse," and "a great deal worse," were collapsed, the latter two having very few responses). Analyzing all respondents in the true experimental agencies combined, experimental group respondents more often said "great improvement" (41% vs. 28% for the control group) and less often rated matters as same or worse (28% vs. 40%). The significance of this difference was .11. The result held in HD and LF (nonsignificant for HD, p = .095 for LF) while it was reversed for OTT (comparison group respondents more often rated things as greatly improved). In MM, the percentages in the three categories for the experimental and control groups were almost the same. Separate analysis of birth parent responses yields similar nonsignificant patterns.

Child Outcomes. A series of 35 yes-no child behavior questions were used to derive four scales (caretaker interview, question 57):

  • Proportion of 25 negative child behaviors,
  • Proportion of 10 positive child behaviors,
  • Number of four child aggression items yes (these items were among the negative behaviors), and
  • Proportion of five school problem items yes (also among the negative behaviors).

These scales were analyzed both for all respondents and for birth parents separately. In the all respondents analysis, no significant differences were found between experimental and comparison groups in any of the agencies. For birth parents, there was one significant difference: in OTT, average scores for child aggression were lower in the experimental group than in the comparison group (p < .05).

Even more than in the case of parental functioning, issues arise in the interpretation of child behavior as an outcome of these services. Most of the services of programs like HomeRebuilders are directed to parents. Therefore, the effects on child behavior are necessarily indirect and might be expected to be relatively small.

Summary. Given the large number of comparisons examined, little can be made of the few significant differences noted above. Nonsignificant differences often favored the comparison group. There is no conclusive evidence in the interviews of caretakers that the HomeRebuilders group had better child and family functioning than the comparison group.

View full report

Preview
Download

"appa.pdf" (pdf, 120.49Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"appb.pdf" (pdf, 623.25Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®

View full report

Preview
Download

"appc.pdf" (pdf, 3.4Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®