Evaluation of Family Preservation and Reunification Programs: Final Report - Volume One. 2.4 Data Collection Activities

12/01/2002

Data collection began with a baseline interview as soon as possible after families were randomly assigned to either group. At that time, we attempted to interview the investigating worker handling the case (if the case originated from an investigator), the caretaker, and the caseworker assigned to the case. The caseworker was also asked to report on all contacts with the family during the time services were provided. At the completion of family preservation services or at a comparable time for cases receiving regular services, we interviewed the caretaker and the caseworker again. One year after enrollment, we conducted a followup interview with the caretaker. In addition to these interviews, we collected data from staff at the participating agencies. Administrative data were collected on individual cases up to eighteen months after random assignment. Table 2-3 shows the data collection status of the study's various questionnaires with agency staff.

The Staff Survey was a seven-page self-administered questionnaire designed to obtain a profile of staff at the participating agencies and information on their attitudes and opinions about family preservation services. The questionnaire was mailed to all staff who potentially could have a case in the study. A concerted effort was made to obtain questionnaires from investigating workers and workers in public and private agencies who had study cases. At most sites, this included all the workers at private agencies that provided family preservation services, any workers in family preservation units in public agencies, and workers in units of public agencies that provided in-home and foster care services. In addition to investigating workers and the workers to whom actual cases were assigned, those workers' supervisors were also asked to complete the survey. The response rate for workers completing staff questionnaires for staff with cases in the study was 90 percent in Kentucky, 76 percent in New Jersey, 79 percent in Tennessee, and 63 percent in Philadelphia.

The Investigating Worker Questionnaire was a six-page self-administered questionnaire designed to capture information about the investigation of a complaint that led to a referral to family preservation services. Information collected included when and how the complaint was investigated, the nature of the allegation, a description of the home, and problems affecting the household.

Table 2-2
Violations and Minimal Service Cases by County
Kentucky
  Jefferson Fayette Total KY
C E C E C E
Net study cases 162 155 13 19 175 174
Violations 9 -- -- -- 9 --
Minimal service 1 48 -- 5 -- 54
New Jersey
  Camden Burlington Ocean Monmouth Essex Bergen Passaic Total NJ
C E C E C E C E C E C E C E C E
Net study cases 19 39 20 47 29 41 23 25 45 62 20 29 11 32 167 275
Violations 1 -- -- -- 6 -- 3 -- 6 -- 6 -- 2 -- 24 --
Minimal service -- 6 -- 8 -- 5 -- 1 -- 13 -- 3 -- 8 -- 44
Tennessee--Shelby County Philadelphia
  C E   C E
Net study cases 49 98 Net study cses 144 209
Violations 3 -- Violations 5 --
Minimal service 1 10 Minimal service 4 67

Table 2-3
Caseworker Response Rates
  Kentucky New Jersey Tennessee Philadelphia
Number Percent Number Percent Number Percent Number Percent
Staff Questionnaires
Staff questionnaires mailed 215   344   81   334  
Completed staff questionnaires 194 90 262 76 64 79 210 63
Investigating Worker Questionnaires
Investigating questionnaires mailed 212   223   140   353  
Completed investigating workers questionnaires 164 77 119 53 109 78 276 77
Cases with no investigating workers 138   219       8  
Caseworker Interviewers
Initial caseworker interviews fielded 349   442   147   353  
Completed initial caseworker interviews 280 80 388 88 112 76 163 46
Post-treatment caseworker interviews fielded 349   444   147   353  
Completed caseworker post-treatment interviews 326 93 434 98 138 94 250 71
Contact Reports
Cases expecting contact report forms (15) 324   428   140   328  
Number of cases with one or more
Completed contact report forms 235 73 369 86 98 68 210 63

As soon as a case referred by an investigating worker was randomly assigned, we mailed an Investigating Worker Questionnaire to the investigating worker reported on the Random Assignment Form. Investigating workers who did not respond to the initial request received reminder letters and second request mailings. If these requests failed, the site coordinator followed up with the worker in person. The response rate for investigating workers completing the questionnaire was 77 percent in Kentucky, 53 percent in New Jersey, and 78 percent in Tennessee. Not all cases were referred by investigating workers in Kentucky and New Jersey; ongoing workers referred 39 percent of the Kentucky cases and 50 percent of the New Jersey cases. All cases in Tennessee were to be referred by investigating workers. However, 5 percent of the cases (7 cases) did not have an investigating worker identified. All cases in Philadelphia were referred by investigating workers. Two percent did not have an investigating worker identified.

The Caseworker Interview was conducted by the Westat Telephone Research Center (TRC). The TRC attempted to conduct an initial and post-treatment interview with the caseworker for each case that was randomly assigned. The initial caseworker interview was to be completed within two weeks of random assignment. If the referring worker was an ongoing caseworker, telephone interviewers attempted to interview him or her as soon as possible. If the referring worker was an investigating worker and the case was a control case, Westat's site coordinator tracked how quickly the investigating worker transferred the case to an on-going unit. (16) If the site coordinator did not get a response from the worker within 10 working days, the investigating worker was identified as the caseworker to be interviewed for the baseline interview, and TRC interviewers had an additional 5 days to obtain the initial interview. This procedure was instituted because some investigating workers did not immediately transfer their cases, which created difficulties in reaching caseworkers within the two-week time frame.

The telephone interviewers experienced some difficulty successfully reaching and interviewing caseworkers during the study's time period, especially the initial caseworker interview period. The response rate for completed initial caseworker interviews was 80 percent in Kentucky, 88 percent in New Jersey, 76 percent in Tennessee, and 46 percent in Philadelphia. The response rate was lower in Philadelphia due to cases not being assigned a caseworker within the initial interview period.

The post-treatment caseworker interview was scheduled to occur at the same time as the post-treatment caretaker interview, that is, at the end of family preservation services or at a comparable point for control group cases. In both the initial and post-treatment interviews, the caseworker was asked to describe the household, including all household members and their relationships to the children mentioned in the complaint; the condition of the home when visited by the caseworker; problems affecting the caretaker and other household members; and an assessment of the children's well being. At the post-treatment interview, the caseworker was asked about services provided and was asked to assess whether the goals for the case were met. If the caseworker had not completed a staff survey questionnaire at the time of the post-treatment interview, the telephone interviewer attempted to ask the staff survey questionnaire questions at the conclusion of the post-treatment interview. The response rate for completed post-treatment caseworker interviews was 93 percent in Kentucky, 98 percent in New Jersey, 94 percent in Tennessee, and 71 percent in Philadelphia. Data on completion of caseworker interviews by county are shown in Table 2-4.

Caseworker Contact Reports were to be completed by all caseworkers for each face-to-face contact with a family member during the time period designated for family preservation services. These forms were one-page checklists on which the workers indicated the services delivered at each contact. The forms capture information on concrete services and the content of counseling (e.g., parenting practices, anger management). For cases assigned to family preservation services, the caseworkers were expected to complete these forms from the time the case was first assigned to them through the end of services. Caseworkers with control cases were expected to complete forms for a comparable time period.

Each time a caseworker received another study case (after the first one), Westat mailed the caseworker a letter of notification. This letter identified the case and informed the caseworker that contact reports were to be completed for it, starting immediately. Caseworkers were instructed to complete the reports when a contact was made and to mail them to Westat at least once a week. Each participating caseworker was mailed a supply of contact report forms and postage-paid return envelopes. When it was time to stop completing reports for a case, Westat sent a letter notifying the caseworker. If no completed forms were received, the caseworker was asked to confirm that there were no in-person visits. Letters were sent to workers to obtain this confirmation. In addition, delinquency reports were sent to site coordinators who in turn contacted caseworkers to remind them to complete the form. Contact reports were received for 73 percent of Kentucky cases, 86 percent of New Jersey cases, 69 percent of Tennessee cases, and

Table 2-4
Caseworker Interview Completion Rates by County
Kentucky  
  Jefferson Fayette Total KY
C E C E C E
% % % % % %
Net study cases 162 155 13 19 175 174
Initial interviews 138 120 6 16 144 136
Post-treatment interviews 157 147 4 18 161 165
Both interviews 136 119 3 16 139 135
New Jersey
  Camden Burlington Ocean Monmouth Essex Bergen Passaic Total NJ
C E C E C E C E C E C E C E C E
% % % % % % % % % % % % % % % %
Net study cases 19 39 20 47 29 41 23 25 45 62 20 29 11 32 167 275
Initial interviews 16 35 16 45 21 40 17 24 39 55 19 23 9 29 137 251
Post-treatment interviews 19 39 20 47 28 41 23 25 42 60 19 28 11 32 162 272
Both Interviews 16 35 16 45 21 40 17 24 37 55 18 22 9 29 134 250
Tennessee- Shelby Philadelphia
  C E   C E
  % %   % %
Net study cases 49 98 Net study cases 144 209
Initial interviews 46 66 Initial interviews 50 113
Post-treatment interviews 48 90 Post-treatment interviews 99 151
Both interviews 46 66 Both interviews 48 112

60 percent of Philadelphia cases. These response rates are based on only those cases for which we expected a contact report. Caseworkers returned letters indicating that no in-person visits were held for 7 percent of the Kentucky cases, 3 percent of the cases in New Jersey, 5 percent of the Tennessee cases and 7 percent of Philadelphia cases. All experimental cases where workers indicated there was no contact were minimal service cases.

Caretaker Interviews were conducted at three points in time. Data collection began with a baseline interview soon after random assignment in order to get an accurate picture of the household just as services began. A Westat field interviewer attempted to interview the person designated as the caretaker on the random assignment form within two weeks of random assignment. During this interview, the caretaker was asked to enumerate and describe all members of the household and to answer questions about the functioning of the household and parenting philosophies and practices. A second or post-treatment interview was conducted at the time family preservation services ended, or a comparable time period for control cases. The post-treatment interview asked questions about the family's makeup and functioning similar to those in the initial interview, as well as additional questions about the services received. A final followup interview with the caretaker was also attempted one year from the random assignment date. The final interview was designed to obtain information similar to that in the initial and post-treatment interviews to measure change over time.

As shown in Table 2-5, the response rate for completed initial caretaker interviews was 89 percent in Kentucky, 74 percent in New Jersey, 80 percent in Tennessee, and 72 percent in Philadelphia. The response rate for completed Post-Treatment Caretaker Interviews was 84 percent in Kentucky, 78 percent in New Jersey, 80 percent in Tennessee, and 74 percent in Philadelphia. For the Follow-up Interview, response rates showed a decrease to 71 percent in Kentucky, 62 percent in New Jersey, 75 percent in Tennessee, and 64 percent in Philadelphia. Successfully completing the caretaker interviews was a data collection challenge for a variety of reasons. The main difficulties included the caretaker not having a telephone number and the mobility of the caretakers. Overall, refusals were rather low: 5 percent at initial, 3 percent at post-treatment, and 4 percent at followup in Kentucky; 6 percent at both initial and post-treatment, and 7 percent at followup in New Jersey; 5 percent at initial and 6 percent at both post-treatment and 6 percent at followup in Tennessee, and 5 percent at initial, 3 percent at post treatment, and 3 percent on followup in Philadelphia. Another reason for noncompletion of interviews was that families could not be located. Table 2-6 shows caretaker interview completion rates by county.

Table 2-5
Data Collection Status for Caretaker Interviews
  Kentucky New Jersey Tennessee Philadelphia
Number Percent Number Percent Number Percent Number Percent
Initial Interviews
Number of cases fielded 349   442   147   353  
Total completed 311 89 328 74 117 80 255 72
Refusals 16 5 29 6 8 5 18 5
Other reasons for closure 22 6 91 20 22 14 80 23
Post-treatment Interviews
Number of cases fielded 349   442   147   353  
Total completed 294 84 344 78 117 80 261 74
Refusals 11 3 26 6 9 6 12 3
Other reasons for closure 44 13 75 17 21 14 80 23
Follow- up Interviews
Number of cases fielded 349   442   147   353  
Total completed 249 71 274 62 110 75 225 64
Refusals 13 4 30 7 10 6 11 3
Other reasons for closure 87 25 138 31 27 19 117 33
Table 2-6
Caretaker Interview Completion Rates by County
Kentucky
  Jefferson Fayette Total KY
C E C E C E
% % % % % %
Net study cases 162 155 13 19 175 174
Initial interviews 146 139 9 17 155 156
Post-treatment interviews 136 134 10 14 146 148
Followup interviews 115 122 4 8 119 130
All three interviews 115 109 3 8 118 117
New Jersey
  Camden Burlington Ocean Monmouth Essex Bergen Passaic Total NJ
C E C E C E C E C E C E C E C E
% % % % % % % % % % % % % % % %
Net study cases 19 39 20 47 29 41 23 25 45 62 20 29 11 32 167 275
Initial interviews 12 25 17 35 21 32 17 15 36 43 17 23 10 25 130 198
Post-treatment interviews 14 29 15 39 22 30 18 18 36 48 18 24 11 22 134 210
Followup interviews 9 22 11 31 18 25 13 13 30 32 15 19 11 25 107 167
All three interviews 4 17 8 26 14 19 11 8 23 26 14 16 10 18 84 130
Tennessee-Shelby Philadelphia
  C E   C E
% %   % %
Net study cases 49 98 Net study cases 144 209
Initial interviews 37 80 Initial interviews 107 156
Post-treatment interviews 37 80 Post-treatment Interviews 113 148
Followup interviews 36 74 Followup interviews 90 135
All three interviews 28 61 All three interviews 70 102

View full report

Preview
Download

"report1.pdf" (pdf, 770.39Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®