Evaluation of Family Preservation and Reunification Programs: Final Report - Volume Two. 5. Attrition Analysis: Caretaker Interviews

Longitudinal studies almost always encounter sample attrition. Not all respondents will be interviewed at all points in time. In this study, some caretakers responded to all three interviews, some caretakers responded to the initial interview but not the post-treatment or followup interview, and some not interviewed earlier were interviewed later. Numbers of cases with caretaker interviews at each of the three points in time are shown in Table 5-1. Percentages are shown in Table 5-2.

Table 5-1
Counts of Cases for all Possible Combinations of Caretaker Interviews Completed
  Kentucky New Jersey Tennessee Philadelphia
C
%
E
%
Total C
%
E
%
Total C
%
E
%
Total C
%
E
%
Total
Time 1 only 9 5 14 5 9 14 2 3 5 7 20 27
Time 2 only 2 1 3 4 15 19   1 1 4 9 13
Time 3 only 5 5 10 4 5 9 2 7 9 1 6 7
Time 1 and time 2 34 28 62 34 46 80 5 14 19 25 22 47
Time 1 and time 3 4 6 10 7 13 20 2 2 4 5 12 17
Time 2 and time 3 2 2 4 12 19 31 4 4 8 14 15 29
All three interviews 108 117 225 84 130 214 28 61 89 70 102 172
Totals 164 164 328 150 237 387 43 92 135 126 186 312

 

Table 5-2
Caretaker Interviews Completed as a Percentage of Net Study Cases
  Kentucky New Jersey Tennessee Philadelphia
C
%
E
%
Total C
%
E
%
Total C
%
E
%
Total C
%
E
%
Total
Net Study Cases 175 174 349 167 275 442 49 98 147 144 209 353
Time 1 only 5.1 2.9 4.0 3.0 3.3 3.2 4.1 3.1 3.4 4.9 9.6 7.6
Time 2 only 1.1 0.5 0.9 2.4 5.5 4.3   1.0 0.7 2.8 4.3 3.7
Time 3 only 2.8 2.9 2.9 2.4 1.8 2.0 4.1 7.1 6.1 0.7 2.9 2.0
Time 1 and time 2 19.4 16.1 17.8 20.4 16.7 18.1 10.2 14.3 12.9 17.4 10.5 13.3
Time 1 and time 3 2.3 3.4 2.9 4.2 4.7 4.5 4.1 2.0 2.3 3.5 5.7 4.8
Time 2 and time 3 1.1 1.1 1.1 7.2 6.9 7.0 8.2 4.1 5.4 9.7 7.2 8.2
All three interviews 61.7 67.2 64.5 50.3 47.3 48.4 57.1 62.2 60.5 48.6 48.8 48.7

Of the cases randomly assigned, the proportion of respondents who completed both the initial and followup interviews ranged from approximately 53 percent to 67 percent. Of primary concern here is that those who did not complete the followup interviews might vary in systematic ways from those who did, thus potentially affecting any analyses of change over time that rely on the interview data. Several steps were taken to examine the sample attrition for differences in those who did not see the study through to the end, particularly in regard to whether there were differences between the experimental and control groups with respect to who completed the followup interview.

Starting with the sample characteristics for those who responded to each of the interviews at each point in time, no significant differences were found in the distributions of the following characteristics: respondent's age, youngest child's age, oldest child's age, number of persons in household, number of adults in household, or number of children in household.

In addition to looking at demographic characteristics, we examined initial interview responses on the family and child functioning scales that were used as primary outcome variables. (97) This was done to detect whether the group that was analyzed in our change analysis was functioning better or worse at the outset compared to the whole sample of cases that were interviewed at the outset. Those who completed the initial interview but did not complete the followup interview were compared to those who completed the initial interview and the followup interview on scale measures at the initial interview. (98) Items on which there were significant differences between those who responded to the initial but not to the followup interview and those who responded to both interviews are reported in Table 5-3.

For those caretaker scale comparisons indicating significant differences between those who remained in the interview sample through the followup period and those who did not, t-tests were conducted to assess differences between experimental and control groups for that particular scale at the time of the initial interview. For example, in Kentucky the cases where respondents did not complete the followup interview were analyzed for differences between experimental and control groups in reports of average child aggression at the time of the initial interview. The

Table 5-3
Differences in Initial Family and Child Functioning Scales
  Completed initial but not followup interview Completed initial and followup interview p
N Mean N Mean
Kentucky
Child aggression 76 1.18 235 1.53 .005
Positive child behaviors 69 .75 228 .70 .02
New Jersey
Negative life events 93 .08 234 .12 .007
Stolen things or arrested 94 .41 234 .58 .02
Tennessee
Child problems 24 1.45 93 2.32 .01
Philadelphia  
Caretaker depression 74 1.25 189 .97 .03
Note: Means represent average scores on the scales at the time of the initial interview (i.e. at the outset of the study).

results were not significant, thus, while those who did not complete the followup interview appear to have reported a lower proportion of child aggression problems at the outset than those who did complete the followup interview, this did not occur differentially for experimental and control groups. None of the t-test comparisons for the items listed in Table 5-3 revealed any significant differences in the average initial scores for the experimental and control groups.

In summary, of the 68 comparisons (4 states, 17 for each state) on initial levels for the family and child functioning scales of the group completing both initial and followup interviews with those completing initial interviews but not followup, only five showed significant differences in means at the initial interview. All five of these measures indicate lower initial functioning of those who were interviewed at both points in time, but none of the comparisons of experimental and control groups on initial levels of these measures were significant.

In conclusion, there is no substantial evidence that attrition resulted in an analytic sample that is unrepresentative of the initial interview sample.

Endnotes

97. There were 17 scales in all: positive life events, negative life events, life events depression, economic functioning, punishment, child aggression, school problems, child withdrawn, stolen things or arrested, child substance abuse, child problems, negative child behaviors, positive child behaviors, household condition, depression, positive child care practices, and negative child care practices.

98. Respondents were categorized regardless of completion of post-treatment interview. Furthermore, those who did not complete an initial interview were excluded all together as information about functioning at the outset of the study was unavailable. The proportion of net study cases without initial interview data and thus excluded from these analyses are as follows: 5 percent in Kentucky, 13 percent in New Jersey, 12 percent in Tennessee, and 14 percent in Philadelphia.

View full report

Preview
Download

"report2.pdf" (pdf, 978.24Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®