Evaluation of Family Preservation and Reunification Programs: Final Report - Volume Two. 3.11 Matching of Services to Problems


The analysis to this point has examined the effects of services in an undifferentiated way, by looking at the relationship between the amount of services and outcomes, by either comparing the outcomes of the experimental and control groups or, in the last section, examining the relationship between amount of services as determined by the number of contacts and outcomes. But it is possible that the services provided were not responsive to the particular problems of families. To the extent that this is the case, it would explain the relatively small effects of services on outcomes. Furthermore, an examination of specific problems and specific services might reveal effects that are obscured in the global analyses presented thus far.

We explored this possibility in a limited way by looking at three prominent problems experienced by families in the study, financial difficulties, problems with discipline of children, and depression. After identifying families with these problems, we determined the extent to which services provided might have addressed these issues, the extent of "match" between problems and services provided. Finally, we determined whether there was a relationship between the extent of match of services and problems with outcomes. We conducted the analyses only on cases in the family preservation group in the three Homebuilders states for which we had interviews at the beginning and end of service and contact forms giving us information on services provided. We limited the sample to the Homebuilders group so as to look at the implementation of a particular, well defined model, thus limiting extraneous variance due to variation in approach. It was thought that within that group we would most likely find a match between problems and services. We also believed that the contact form data, on which this analysis depended, was more complete and of a higher quality for the experimental group. Furthermore, in general, control group cases did not receive a high enough volume of services to reveal a match.

We combined the samples from the three states, a total of 292 families with 886 children. Women were the caretakers in 89 percent of these families and the caretakers were, on average, 36 years old. Forty-eight percent were African American, 47 percent were white.

Economic problems were determined from responses on the caretaker interviews to questions about difficulties paying rent, paying electric and heating bills, buying food, and buying clothes for the children. Any family responding positively to any one of these four questions at the initial interview was classified as having economic problems (n = 157, 54%). The presence of disciplinary problems was determined from seven questions on the initial caretaker interview. (54) Again, if any one of these questions was answered affirmatively, the case was classified as having disciplinary problems (n = 221, 76%). Caretaker depression was determined from the SCL-90 depression scale. Individuals with scores higher than the median (determined separately for men and women) were categorized as having some problems with depression.

Services specific to particular problems were determined from the contact forms completed by family preservation caseworkers. Services responding to economic problems were providing emergency cash or paying bills; buying food; helping to find housing; providing clothing, furniture, or supplies; and discussions of money management. Services directed at disciplinary practices were discussions of discipline of children, child's anger management, dealing with violence in the family, caretaker interaction with the child, supervision of children, and child development. Services directed at caretaker depression were discussions of depression, other caretaker emotional problems, social skills, and adult companionship. The extent of service response to particular problems was calculated as the simple count of the number of times caseworkers checked a particular item on the contact forms submitted for the family.

There are a number of limitations in this procedure. First, we looked at only three problem areas, areas that we thought we could identify relatively easily. Further, the measurement of service response is clearly not ideal, service data were not constructed in a way that would make them straightforwardly parallel to problems, so the development of service measures in this analysis is quite post-hoc. In addition, obviously caseworkers may have chosen, for good reason, not to respond to a particular problem, perhaps because another problem was more pressing or more tractable, so that the failure to respond to a particular problem should not be viewed as an indication of the failure of casework in the case. There was also considerable overlap of these problems (83 families reported all three problems at the initial interview, 82 reported two of them, while 24 reported none of them). Despite these limitations, tendencies toward the matching of services and problems should show up in the data, though perhaps not as prominently as would have been the case had data collection been explicitly directed at exploring these issues.

To determine the match between problems and services, we calculated the average number of times a service was provided to cases with the problem (the number of contact forms recording the service) and compared that to the average number of times the service was provided to cases that did not have the problem. The results are shown in Table 3-16.

Families with economic problems significantly more often than families without such problems received three of the five services identified as responding to these problems. We conclude that there is moderate evidence of a match between problems and services in the economic area. Families with discipline problems received two of five services significantly more often than those without these problems, some indication of a match. As to depression, caseworkers significantly more often discussed depression with caretakers with high levels of depression than with those with lower levels. For the other three services identified as possibly responding to caretaker depression there were no significant differences. Discussion of "other caretaker emotional problems" occurred more often with those with higher levels of depression, although the difference is not significant. We conclude, therefore, that there is some match between problems and casework response, a match that might have shown itself more strongly had measurement of services been designed to reveal it.

Table 3-16
Match Of Services With Problems (Mean number of times service provided)
  Cases with the problem Cases not having the problem
Economic problems, n = 157 134
Emergency cash or paying bills 0.57** 0.23
Buying food 1.26** 0.63
Helping to find housing 0.37 0.36
Clothing, furniture, supplies 0.92 0.66
Money management discussion 2.05** 0.93
Discipline problems, n = 221 68
Services, discussion of:
Discipline of child 7.35** 5.08
Child anger management 4.98* 3.33
Family violence 2.53 1.73
Caretaker interaction with child 6.32 5.82
Supervision of children 3.45 3.76
Depression, n = 152 140
Services, discussion of:
Caretaker depression 2.83** 1.43
Other caretaker emotional problems 2.37 1.80
Adult companionship 1.32 0.99
Social skills 2.42 2.85
* p < .05, ** p < .01.

Effect of Match of Services to Problems on Outcomes.

We next attempted to determine whether, for cases experiencing each of the three problem categories, service responses specific to the problem made a difference in outcome. The outcomes we examined were improvements in the specific problem at the time of the post-treatment caretaker interview and subsequent placement and maltreatment. Because families in the experimental group were not randomly assigned to varying levels of specific kinds of services, it can be assumed that there are selection biases operating in determining levels of service. Hence, for the examination of effects on the level of the problem at the post-treatment interview, two-stage least squares techniques were used in which the provision of specific services was modeled in the first stage and the effects of services on outcomes were determined in the second stage. Demographic characteristics were included in both stages and the level of the problem at the initial interview was included in the second stage. A variable reflecting the caretaker's response in the post-treatment interview to a question on whether additional services were needed was also included in both stages. First stage instruments were chosen based on their prediction of levels of service and lack of association with the outcome variables. In all of the analyses the instruments were jointly significant in the prediction of levels of service.

We report here on the results of the second stage of these analyses, the determination of the effect of specific services on problem level at the post-treatment interview. For all three problems, as would be expected, the initial level of the problem was significantly and positively related to the post-treatment level. For the group experiencing economic problems, of the five services thought to respond to the problem, only one was significantly related to post-treatment problem levels, the provision of cash assistance was positively related to level of economic problems post-treatment. That is, the more cash assistance provided, the higher the levels of economic problems. It is likely that this seemingly contrary finding simply reflects the fact that families with considerable economic difficulties are more likely to get cash assistance but are also more likely to continue to experience those problems. The variable reflecting need for additional concrete services was also positively related to post-treatment levels of economic problems.

For the group with discipline problems, of the five services in the second stage equation, two were significantly related to the level of the problem at post-treatment. The higher the level of discussion of discipline of children, the lower the level of the problem while the greater the discussion of child anger management, the higher the level of the problem. For cases with above median levels of depression, two of the four services thought to respond to the problem were significantly related to level of the problem at post treatment. More discussion of depression in caseworker contacts was related to higher levels of depression post-treatment while more discussion of adult companionship was related to decreased post-treatment depression.

The sparseness of positive findings in this analysis leads us to conclude that there is little evidence here of positive effects of concrete or clinical services on these three problem areas.

To examine the effects of specific services on placement and maltreatment following entry into the study, hierarchical linear modeling was used in order to account for the fact that we have multiple children in some families and their outcomes are not independent. (55) To deal with selection issues, predicted values of services, determined from the first stage of the 2SLS analyses, were entered at level two (the family level) and these terms were used to determine the effects of services on outcomes. Again, separate analyses were done for each problem group.

For the economic difficulties subgroup, two of the five problem specific services were significantly and negatively related to the likelihood of subsequent maltreatment: provision of clothing/furniture/supplies and housing assistance. Cash assistance and provision of clothing/furniture/supplies were significantly and negatively related to the likelihood of subsequent placement. For families with problems in discipline of children, none of the six services were related to subsequent maltreatment, while discussions of violence in the family were related to a decrease in likelihood of placement and discussions of child anger management were related to an increase. In families in which the caretaker was measured as having higher than median levels of depression at the beginning of service, none of the problem specific services were related to subsequent maltreatment. However, discussion of depression was related to an increase in likelihood of placement while discussions of social skills and of companionship/friendship were related to a decrease in placement.

No consistent patterns emerge in the analysis of the effects of specific services on subsequent maltreatment and placement. There is some indication that within the economic problems group, services directed at these problems have some beneficial effects on these outcomes. Within the discipline problems and depression groups, results are mixed, some services are related to increases in subsequent maltreatment and placement while others are associated with decreases. No clear conclusions can be drawn.

View full report


"report2.pdf" (pdf, 978.24Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®