Evaluation of Family Preservation and Reunification Programs: Final Report: Volume 1: Implementation

2. Implementation

[Main Page of Report | Contents of Report]

Contents

• 2.1 Site Selection and Recruitment
• 2.2 Negotiations
• 2.3 Random Assignment and Case Enrollment Status
• 2.4 Data Collection Activities
• 2.5 Lengths of Time from Random Assignment to the Interviews
• 2.6 Administrative Data
• 2.7 Maintaining Study Integrity

Endnotes

Designing a rigorous experimental study is only the first step; its implementation is a formidable task. Convincing administrators to subject their staff and programs to intense scrutiny is the first challenge, followed by implementing the evaluation in an ongoing service delivery environment. Negotiations required repeated meetings with administrative, supervisory, and front-line staff. We had to establish a dialogue to foster open communication in which fears, expectations, and study requirements could be discussed. Implementation required continual communication with site personnel. This communication included periodic site visits, monthly written reports to sites about the status of cases enrolled in the study, and a site coordinator stationed at each site to aid in the daily data collection effort. This chapter presents an overview of site selection, negotiations, and the data collection effort. Further description of site-specific implementation efforts in Kentucky, New Jersey, Tennessee, and Philadelphia are presented in Chapters 3, 4, 5, and 6.

2.1 Site Selection and Recruitment

The site selection process began with a discussion with personnel in potential sites of the issues and criteria surrounding site selection. The task of applying these criteria to real programs began a process of reconciling the differences between our hope of finding optimal sites and program and practice realities. We initially identified potential states and counties for the study through review of state plans, contacts with experts in the field, reviews of the literature, and previous studies conducted by the research team. Based on this review, we contacted 26 states and asked them about their family preservation and reunification programs with respect to our criteria for selection.

A list of programs and counties contacted is presented in Appendix A. Results of the telephone conversations with these sites were presented in the Review of Family Preservation and Reunification Programs. Based on responses to the telephone conversations and extensive discussion among research team members and the advisory panel, we eliminated a number of states or particular counties within states from consideration.

To obtain more detailed information about states, site visits were necessary. As we were unable to conduct site visits to all identified states, we established two levels of site visits. The first level targeted states that had some of the best and most mature programs in the country: Kentucky, Michigan, Missouri, New Jersey, and Washington. Project staff conducted 3- to 5-day site visits at the state level and in those local jurisdictions that might be included in the study. The visits included meetings with administrative, supervisory, and casework staff at the state and local levels of the public child welfare agency. We also conducted interviews with administrators and caseworkers of the local family preservation agencies. Through the interviews, we gathered information about family preservation services and the context in which the services were being delivered. States' interest in the study and their ability to meet selection criteria were also explored. We then conducted further site visits in Tennessee, Oregon, California, Florida, New York, and Ohio. ⁽¹⁰⁾ Our emphasis was on selecting quality programs and those with characteristics useful for the development of knowledge (e.g., serving clientele with substance abuse problems). It was decided to evaluate three programs that reported using relatively "pure" versions of the Homebuilders model of service. The sites selected were Memphis, Tennessee; Louisville, Kentucky; and seven counties in New Jersey. These three sites met the original criteria set forth by contract requirements and also incorporated the other issues identified as important. All three sites identified a targeting problem and were interested in implementing targeting strategies, had a long and positive history of providing quality Homebuilders programs, had a limited number of providers, and had adequate support for the program in the responsible public agency. Also, all the sites identified a pool of families who were eligible for the services but not receiving them and had sufficient numbers to reach study sample size requirements (or agreed to continue the study for more than a year, if necessary).

For the fourth site, our efforts turned to identifying a non-Homebuilders family preservation program, which was well defined and able to articulate its goals and objectives. While the study team visited Philadelphia to explore its reunification programs, its family preservation program was also presented as an option. The program had many interesting and policy relevant elements. The family preservation programs in Philadelphia are based on specialization, and the county has a strong focus on serving families with substance abuse problems. Philadelphia County represents a site in which the goal of family preservation services is defined more broadly than placement prevention, allowed comparison of family preservation services to less intensive in-home services, and has some agencies with an explicit focus on substance abuse. These criteria lent themselves to the selection of Philadelphia for the fourth site.

[Go To Contents]

2.2 Negotiations

Negotiations began during the initial site visits. Discussions with staff focused on obtaining information on the state program and system while providing information to the state about the study. Site visitors needed to determine, as quickly as possible, if states were not interested in participating. Also, we had to establish site flexibility in working within the study guidelines and adhering to rigorous data collection methods early in the negotiation process. Negotiations always began at the state level to obtain permission from the child welfare commissioner or director. Although negotiations were tailored to individual sites, we followed general procedures which entailed numerous meetings with state and local personnel, written permission from the state director of child welfare services, and an agreed-upon detailed work plan delineating target populations, random assignment procedures, data collection plans, and targeting procedures. It was critical to go through a process with state and local agency personnel in which we explored their receptiveness to an experiment, including some alteration in referral procedures and a willingness to fill out our forms and partake in interviews. The most difficult process was working through workers' concerns about withholding services from the control group. Extensive discussions were held about denying services to clients and having a computer make decisions about families' lives. Although many caseworkers never felt totally comfortable with the idea of randomly assigning families to receive either family preservation services or other services, they eventually became resigned to the procedure. Many did come to accept that the experiment was set up to provide services to the same number of families served by family preservation prior to the study and understood that their present systems did not serve all families eligible for family preservation. It was more difficult for them to accept that particular families on their caseloads could not receive a service that they believed to be the best alternative for the families.

Targeting. A major problem that has plagued family preservation programs and their evaluations is targeting. To prevent placements effectively, these programs have been intended for cases in which there is an "imminent risk of placement." Previous studies have indicated that family preservation services are often delivered to families in which placement is not likely. A goal of this evaluation was to address the targeting problem in at least some of the placement prevention programs to be studied so that the programs would have the best possible chance of success on the outcome measure of preventing foster care placement. We selected sites that realized that targeting was an issue and that were interested in developing strategies to improve targeting. We believed that targeting could be improved through removing from the referral pool some of the cases that would not experience placement in the absence of family preservation services or through diverting to the family preservation referral pool some cases that were placed. This might be called screening out the cases that are not at imminent risk of placement and screening in the cases that are going to be placed but can be safely maintained at home. To aid in this process the study team developed a screening tool for local agency personnel responsible for referring cases for family preservation services. The tool provides personnel the opportunity to review their decisions by using a risk index based on factual items such as previous substantiated complaints, more than one maltreated child, previous foster care placements, and the presence of substance abuse. The instrument yields a score, the midrange values of which were thought to suggest referral to funding preservation. A copy of the protocol is in Appendix B. A further discussion on the use of this screening protocol in Kentucky and New Jersey is included in each individual site report in Chapters 3 and 4.

Implementation plans for each site built upon already existing procedures. A written work plan was worked out with each site. A brief description of the plans for each site is presented below.

Kentucky has a statewide program using the Homebuilders model. A statewide coordinator is responsible for developing uniform selection criteria, training of and contracting with providers, and overseeing the program. The study was conducted in Louisville, where there is a single family preservation program provider, and child abuse and neglect cases are referred from intake or ongoing workers. ⁽¹¹⁾ There was no age limitation on the children included in the experiment. Because family preservation does not serve drug abuse cases unless the caretaker is in treatment, or sexual abuse cases in which the perpetrator is in the home, these cases were excluded from the experiment. Referral to family preservation begins with worker and supervisor approval. A screener reviews all cases referred for family preservation to determine appropriateness of the referral. Based on this process, we asked the screener to use the screening protocol developed for the study. The protocol aided the screener in reviewing the risk level of each case. In addition, all cases for which a court petition was filed were reviewed to determine whether they met family preservation criteria. If they did, they were referred to the screener who decided whether to refer the case for family preservation services. We conducted this review to identify cases that might be diverted from potential placement. A full-time site coordinator in the Louisville office assisted the screener and workers with survey tasks.

New Jersey has a statewide program using the Homebuilders model. As in Kentucky, a state office coordinator is responsible for uniform selection criteria, training of and contracting with providers, and overseeing the program. The study was conducted in seven counties: Bergen, Burlington, Camden, Essex, Monmouth, Ocean, and Passaic. The study population included Division of Youth and Family Service (DYFS) child abuse and neglect and family problem cases referred from intake or ongoing workers. The state had been trying to refocus delivery of family preservation services to families with younger children. Not all counties made this change, so all children under 18 were included in the experiment. Each of the counties has a screener who reviews referrals to make sure necessary information is provided. The screener continued in this role during the experiment. In addition, we asked workers and their supervisors to apply the study screening protocol to all cases being referred to family preservation to review their referral decisions. In some counties, the screening protocol was also used on cases being referred for foster care placement. Two site coordinators were assigned to help screeners and workers across the seven counties.

Tennessee. During the study period, Tennessee had a statewide program using the Homebuilders model. As with the other study sites, a state coordinator was responsible for developing uniform selection criteria, training and contracting with providers, and overseeing the program. The study was conducted in Shelby County. There was only one Homebuilders agency in the county. However, Shelby County is a service rich county in which there were a number of other service options similar to Homebuilders available to families in the control group. The study population included Division of Children Services child abuse and neglect cases referred from intake workers. Only families in which at least one of the referred children was under 13 were accepted into the study.

Prior to the study, caseworkers referred families directly to the Homebuilders program. For the study, two hotline workers served as study screeners. Referral to family preservation began with worker and supervisor approval. The worker then called the designated screener to find out if there was an opening in family preservation. If an opening was available, the screener would contact Westat to obtain a random assignment. A full-time site coordinator in the Shelby office assisted the screener and workers with data collection.

Philadelphia. The Philadelphia family preservation program was not a Homebuilders model program during the time of the study. The program used a service model that was broader than the traditional Homebuilders model. Children were not considered at imminent risk of removal and services were provided for a longer period than Homebuilders services. The state office was responsible for training and program oversight, and the agency specialized FPS section developed selection criteria for referral. Family preservation services were provided by private agencies in a public-private collaboration. The evaluation included three private agencies -- Abraxas Foundation, Tabor Children's Services, and Youth Service, Inc. Family preservation services were provided by Abraxas Foundation and Tabor Children's Services. All three agencies provided non-FPS Services to Children in their Own Home (SCOH) services to families. Cases were referred only from intake workers. Referrals came through a public supervisor who screened cases for FPS.

[Go To Contents]

2.3 Random Assignment and Case Enrollment Status

Random Assignment. Individual referral and random assignment procedures were developed for each site. These procedures built upon existing agency referral procedures to family preservation. In both Kentucky and New Jersey, the screener made random assignment referrals. Random assignment began in May 1996 in Kentucky and in November 1996 in New Jersey and Tennessee. In Kentucky and New Jersey random assignment ended in February 1998, and in Tennessee, random assignment ended in May 1998. In Philadelphia, cases were entered into the study from March 1997 to June 1999.

Cases were referred to the screener, who, depending upon the site, either determined if the case was appropriate for family preservation or merely made sure that space was available. The screener then called Westat for assignment of the case. The Westat assignment clerk asked for some basic information about the case. In most instances random assignment was done while the screener stayed on the telephone. The screener then mailed or faxed the family preservation referral form to provide more details about the case. This form was used to fill in the study's random assignment form. (see Appendix C).

Westat personnel used a computer program to randomly assign the case to either the experimental or control group. For those cases randomly assigned to the experimental group, the

family received family preservation services. For those cases assigned to the control group, the family received other services provided by the agency.

Case Enrollment and Status. Table 2-1 shows the number of cases enrolled by county. A 50/50 experimental/control assignment was planned in Kentucky, and a 60/40 experimental/control assignment in New Jersey and Philadelphia. Tennessee began with a 60/40 experimental/control assignment which changed to 70/30 about six months into the study. The actual proportions assigned to each group fell within the expected range.

Some eligible cases were not referred for random assignment and did not get into the study but did receive family preservation services. Exceptions were granted only with the approval of state officials who reviewed the case and determined whether to bypass the study. The state was asked to report exceptions, but sometimes these cases were only detected during review of agency logs and screener telephone calls. Over the course of the study, there were 5 exceptions in Kentucky, 33 exceptions in New Jersey, and none in Tennessee or Philadelphia.

In Kentucky a total of 358 cases were randomly assigned by the Department of Social Services (DSS), ⁽¹²⁾ 323 in Jefferson County (Louisville) and 35 in Fayette County (Lexington). ⁽¹³⁾ Of these, 9 were determined to be inappropriate referrals and were excluded from the analyses (6 in the experimental group and 3 in the control group). The 9 inappropriate referrals included 3 reunification cases, 4 cases in which the children identified as at risk were out of the home, and 1 case where the custodial parent was incarcerated (in one case the reason for inappropriate referral was not identified). After removing the 9 inappropriate referrals, there were 174 net study cases in the experimental group and 175 net study cases in the control group.

The New Jersey evaluation involved programs in seven counties. A total of 470 cases were randomly assigned from the Department of Youth and Family Services, 288 in the experimental group and 182 in the control group. Of the 470 cases that were randomly assigned, 28 cases were determined to be inappropriate referrals (13 in the experimental group and 15 in the control group). Seventeen of these inappropriate referral cases were reunification cases. The remaining inappropriate referrals included foster care cases, cases with no child at risk in the home, or cases that had previously received family preservation services and were being re-referred for a "booster" session. After removing the 28 inappropriate referrals, there were 275 net study cases in the experimental group and 167 net study cases in the control group. The numbers of cases in each county in New Jersey are too small to allow for separate analyses of data by county, so we combine them in all analyses of this report.

The Tennessee evaluation in Shelby County included 153 cases randomly assigned by the Division of Children's Services (DCS). Of these, six were determined to be inappropriate referrals and were excluded from the analyses (3 in each of the groups). The inappropriate referrals were due to no children under the age of 13 in the home (one case), three reunification cases, and one case with children in foster care. The sixth inappropriate referral was screened out by DCS. After removing the six inappropriate referrals, there were 49 net study cases in the control group and 98 net study cases in the experimental group.

In Philadelphia, 362 cases were randomly assigned, nine of which were inappropriate referrals, five from the control group and four from the experimental group. The nine inappropriate referrals include reunification cases, cases in which the children identified as at risk were out of the home, one case that was already receiving services, and cases from units that were not participating in the study. After removing the inappropriate referrals, there were 144 net cases in the control group and 209 net cases in the experimental group for a total of 353 net study cases.

The basic analysis of differences between experimental and control groups concerned those cases labeled "Net Study Cases." However, in a few cases the group assignment was violated, that is, the group to which a family was assigned was switched. Although cases that were deemed to require family preservation should have been designated as exceptions, we allowed each state 6 "approved violations," that is, the state central office could switch the groups following random assignment, upon application from the local office. Despite the allowance of 6 violations, 9 Kentucky cases were switched from the control to the experimental group, 8 of these switches were approved and 1 additional violation was unapproved. New Jersey had 24 violations, 19 approved and 5 unapproved, 14 percent of the net study cases assigned to the control group. In Tennessee, three cases were switched from the control to the experimental group and in Philadelphia there were five switches. There were no recorded switches from the experimental group to the control group in any of the states.

Some cases in the experimental group were provided minimal services because of refusal by the family to participate, failure of the family to comply with initial expectations of the program, or because the provider agency turned the case back. Turnbacks occurred when family preservation services workers were unable to contact the family or the family did not meet the criteria for service (in a few such cases, children were not considered to be at risk). There were 53 minimal service cases in Kentucky, 5 noncompliance, 18 refusals, and 31 turnbacks. ⁽¹⁴⁾ In New Jersey, 44 cases assigned to the experimental group received minimal services because of refusal (14 cases), noncompliance by the caretaker (7 cases), or because the case was turned back by the family preservation agency (23 cases). Tennessee had 11 minimal service cases because of refusal (4 cases), the DCS worker never followed through (1 case), the family preservation agency turned back the case due to safety issues (3 cases), and children placed in foster care (3 cases). Seventeen of the 52 minimal service cases in Kentucky had at least one caseworker contact. One case had more than 5 contacts. In New Jersey, of the 44 minimal service cases, on 31 (70%) we had at least one contact. Seven of the 31 families had more than 5

contacts. In Philadelphia, there were 67 minimal service cases in the experimental group and 4 in the control group. The distribution of violations and minimal service cases is shown in Table 2-2.

[Go To Contents]

2.4 Data Collection Activities

Data collection began with a baseline interview as soon as possible after families were randomly assigned to either group. At that time, we attempted to interview the investigating worker handling the case (if the case originated from an investigator), the caretaker, and the caseworker assigned to the case. The caseworker was also asked to report on all contacts with the family during the time services were provided. At the completion of family preservation services or at a comparable time for cases receiving regular services, we interviewed the caretaker and the caseworker again. One year after enrollment, we conducted a followup interview with the caretaker. In addition to these interviews, we collected data from staff at the participating agencies. Administrative data were collected on individual cases up to eighteen months after random assignment. Table 2-3 shows the data collection status of the study's various questionnaires with agency staff.

The Staff Survey was a seven-page self-administered questionnaire designed to obtain a profile of staff at the participating agencies and information on their attitudes and opinions about family preservation services. The questionnaire was mailed to all staff who potentially could have a case in the study. A concerted effort was made to obtain questionnaires from investigating workers and workers in public and private agencies who had study cases. At most sites, this included all the workers at private agencies that provided family preservation services, any workers in family preservation units in public agencies, and workers in units of public agencies that provided in-home and foster care services. In addition to investigating workers and the workers to whom actual cases were assigned, those workers' supervisors were also asked to complete the survey. The response rate for workers completing staff questionnaires for staff with cases in the study was 90 percent in Kentucky, 76 percent in New Jersey, 79 percent in Tennessee, and 63 percent in Philadelphia.

The Investigating Worker Questionnaire was a six-page self-administered questionnaire designed to capture information about the investigation of a complaint that led to a referral to family preservation services. Information collected included when and how the complaint was investigated, the nature of the allegation, a description of the home, and problems affecting the household.

As soon as a case referred by an investigating worker was randomly assigned, we mailed an Investigating Worker Questionnaire to the investigating worker reported on the Random Assignment Form. Investigating workers who did not respond to the initial request received reminder letters and second request mailings. If these requests failed, the site coordinator followed up with the worker in person. The response rate for investigating workers completing the questionnaire was 77 percent in Kentucky, 53 percent in New Jersey, and 78 percent in Tennessee. Not all cases were referred by investigating workers in Kentucky and New Jersey; ongoing workers referred 39 percent of the Kentucky cases and 50 percent of the New Jersey cases. All cases in Tennessee were to be referred by investigating workers. However, 5 percent of the cases (7 cases) did not have an investigating worker identified. All cases in Philadelphia were referred by investigating workers. Two percent did not have an investigating worker identified.

The Caseworker Interview was conducted by the Westat Telephone Research Center (TRC). The TRC attempted to conduct an initial and post-treatment interview with the caseworker for each case that was randomly assigned. The initial caseworker interview was to be completed within two weeks of random assignment. If the referring worker was an ongoing caseworker, telephone interviewers attempted to interview him or her as soon as possible. If the referring worker was an investigating worker and the case was a control case, Westat's site coordinator tracked how quickly the investigating worker transferred the case to an on-going unit. ⁽¹⁶⁾ If the site coordinator did not get a response from the worker within 10 working days, the investigating worker was identified as the caseworker to be interviewed for the baseline interview, and TRC interviewers had an additional 5 days to obtain the initial interview. This procedure was instituted because some investigating workers did not immediately transfer their cases, which created difficulties in reaching caseworkers within the two-week time frame.

The telephone interviewers experienced some difficulty successfully reaching and interviewing caseworkers during the study's time period, especially the initial caseworker interview period. The response rate for completed initial caseworker interviews was 80 percent in Kentucky, 88 percent in New Jersey, 76 percent in Tennessee, and 46 percent in Philadelphia. The response rate was lower in Philadelphia due to cases not being assigned a caseworker within the initial interview period.

The post-treatment caseworker interview was scheduled to occur at the same time as the post-treatment caretaker interview, that is, at the end of family preservation services or at a comparable point for control group cases. In both the initial and post-treatment interviews, the caseworker was asked to describe the household, including all household members and their relationships to the children mentioned in the complaint; the condition of the home when visited by the caseworker; problems affecting the caretaker and other household members; and an assessment of the children's well being. At the post-treatment interview, the caseworker was asked about services provided and was asked to assess whether the goals for the case were met. If the caseworker had not completed a staff survey questionnaire at the time of the post-treatment interview, the telephone interviewer attempted to ask the staff survey questionnaire questions at the conclusion of the post-treatment interview. The response rate for completed post-treatment caseworker interviews was 93 percent in Kentucky, 98 percent in New Jersey, 94 percent in Tennessee, and 71 percent in Philadelphia. Data on completion of caseworker interviews by county are shown in Table 2-4.

Caseworker Contact Reports were to be completed by all caseworkers for each face-to-face contact with a family member during the time period designated for family preservation services. These forms were one-page checklists on which the workers indicated the services delivered at each contact. The forms capture information on concrete services and the content of counseling (e.g., parenting practices, anger management). For cases assigned to family preservation services, the caseworkers were expected to complete these forms from the time the case was first assigned to them through the end of services. Caseworkers with control cases were expected to complete forms for a comparable time period.

Each time a caseworker received another study case (after the first one), Westat mailed the caseworker a letter of notification. This letter identified the case and informed the caseworker that contact reports were to be completed for it, starting immediately. Caseworkers were instructed to complete the reports when a contact was made and to mail them to Westat at least once a week. Each participating caseworker was mailed a supply of contact report forms and postage-paid return envelopes. When it was time to stop completing reports for a case, Westat sent a letter notifying the caseworker. If no completed forms were received, the caseworker was asked to confirm that there were no in-person visits. Letters were sent to workers to obtain this confirmation. In addition, delinquency reports were sent to site coordinators who in turn contacted caseworkers to remind them to complete the form. Contact reports were received for 73 percent of Kentucky cases, 86 percent of New Jersey cases, 69 percent of Tennessee cases, and

60 percent of Philadelphia cases. These response rates are based on only those cases for which we expected a contact report. Caseworkers returned letters indicating that no in-person visits were held for 7 percent of the Kentucky cases, 3 percent of the cases in New Jersey, 5 percent of the Tennessee cases and 7 percent of Philadelphia cases. All experimental cases where workers indicated there was no contact were minimal service cases.

Caretaker Interviews were conducted at three points in time. Data collection began with a baseline interview soon after random assignment in order to get an accurate picture of the household just as services began. A Westat field interviewer attempted to interview the person designated as the caretaker on the random assignment form within two weeks of random assignment. During this interview, the caretaker was asked to enumerate and describe all members of the household and to answer questions about the functioning of the household and parenting philosophies and practices. A second or post-treatment interview was conducted at the time family preservation services ended, or a comparable time period for control cases. The post-treatment interview asked questions about the family's makeup and functioning similar to those in the initial interview, as well as additional questions about the services received. A final followup interview with the caretaker was also attempted one year from the random assignment date. The final interview was designed to obtain information similar to that in the initial and post-treatment interviews to measure change over time.

As shown in Table 2-5, the response rate for completed initial caretaker interviews was 89 percent in Kentucky, 74 percent in New Jersey, 80 percent in Tennessee, and 72 percent in Philadelphia. The response rate for completed Post-Treatment Caretaker Interviews was 84 percent in Kentucky, 78 percent in New Jersey, 80 percent in Tennessee, and 74 percent in Philadelphia. For the Follow-up Interview, response rates showed a decrease to 71 percent in Kentucky, 62 percent in New Jersey, 75 percent in Tennessee, and 64 percent in Philadelphia. Successfully completing the caretaker interviews was a data collection challenge for a variety of reasons. The main difficulties included the caretaker not having a telephone number and the mobility of the caretakers. Overall, refusals were rather low: 5 percent at initial, 3 percent at post-treatment, and 4 percent at followup in Kentucky; 6 percent at both initial and post-treatment, and 7 percent at followup in New Jersey; 5 percent at initial and 6 percent at both post-treatment and 6 percent at followup in Tennessee, and 5 percent at initial, 3 percent at post treatment, and 3 percent on followup in Philadelphia. Another reason for noncompletion of interviews was that families could not be located. Table 2-6 shows caretaker interview completion rates by county.

[Go To Contents]

2.5 Lengths of Time from Random Assignment to the Interviews

Table 2-7 shows lengths of time between random assignment and each interview--initial, post-treatment, and followup--as well as the lengths of time between the initial and post-treatment interviews.

Kentucky. For the 311 initial interviews with caretakers, the length of time from random assignment to completion ranged from 1 to 50 days with an average of 12.6 days (s.d. = 8.1 days, 75% within 16 days, 90% within 23 days). For the 280 caseworker initial interviews, the length of time from random assignment to interview completion ranged from 3 days to 75 days with an average time of 17.6 days (s.d. = 9.36 days, 75% within 23 days, 90% within 28 days).

At post-treatment, 294 caretakers were interviewed, and the length of time from random assignment to interview completion ranged from 24 days to 111 days with an average of 44.8 days (s.d. = 10.5 days, 75% within 49 days, 90% within 58 days). Three hundred twenty-six caseworker interviews were completed in an average of 51 days (s.d. = 14.3 days) after random assignment, with a completion time ranging from 10 days to 142 days (75% within 55 days, 90% within 68 days). For 3 of the cases where services were terminated early (10, 14, and 17 days after random assignment), post-treatment interviews were completed at the time of termination, thus the minimum of 10 days.

With a goal of completing initial interviews within two weeks of the referral date, the intent was to capture each family's situation at the inception of family preservation or regular services. However, initial interviews with caretakers took an average of over 12 days to complete and initial interviews with caseworkers took an average of over 2 weeks to complete. No significant differences were found between control and treatment groups with regard to the time from random assignment to completion of any of the interviews.

As already noted, the first interview was to be conducted within two weeks of the referral and the second interview was to be conducted at the end of service provision or a comparable time. Therefore, it is expected that for those cases where both interviews were completed, approximately four weeks should have passed between the dates of the first and second interviews. Two hundred and eighty-seven caretakers completed both the first and second interviews, and the average length of time between these interviews was 32.3 days (s.d. = 10.3 days, 75% within 37 days between interviews, 90% within 45 days between interviews). For the 274 caseworkers who completed both interviews, the average length of time

between these interviews was 33.2 days (s.d. = 13.3 days, 75% within 38 days between interviews, 90% within 49 days between interviews). ⁽¹⁷⁾

Followup interviews were completed by 219 caretakers an average of 379.9 days after random assignment (s.d. = 14.9, 75% within 387 days, 90% within 401 days). The difference between experimental and control groups with respect to the length of time between random assignment and followup interviews was not significant.

New Jersey. On average, the 325 initial interviews with caretakers were completed 15.3 days (s.d. = 8.5 days) following random assignment (for three cases we do not have the date of the interview). The range in time to completion was 1 to 50 days (75% were completed in 20 days, 90% in 27 days). As in Kentucky, it is not possible to consider the first interview as representing the situation at the inception of family preservation or regular services. In the case of the family preservation cases, these interviews were conducted, on average, two weeks into a four-week intervention. For the 388 caseworker initial interviews, the mean time to completion was 15.6 days (s.d. = 8.1) with a minimum of 2 and a maximum of 40 (75% within 21 days, 90% within 28 days).

For the 344 caretaker post-treatment interviews, the average length of time between random assignment and interview was 52.1 days (about 7 and a half weeks, s.d. = 11.8 days) with a minimum of 33 and a maximum of 116 days (75% within 58 days, 90% within 68 days). For 434 caseworker post-treatment interviews the average was 56.0 days (s.d. = 14.7) with a minimum of 13 and a maximum of 115 (75% within 63 days, 90% within 75 days, interviews on cases that terminated early were sometimes conducted before the end of the 28 day service period, hence the minimum of 13). There were no significant differences between the experimental and control groups in the average lengths of time to interview except for the caseworker post-treatment interview. For the control group, this interview was conducted an average of 58.7 days after random assignment while the average for the experimental group was 54.4 days (p = .003). ⁽¹⁸⁾

Both caretaker interviews were completed in 292 cases, with an average of 37.3 days between interviews (s.d. = 11.2, 75% with not more than 43 days between interviews, 90% not more than 49 days). Three hundred eighty-four caseworkers completed both interviews with an average of 39.8 days between interviews (s.d. = 15.0). ⁽¹⁹⁾

Followup interviews were completed by 218 caretakers an average of 383.5 days after random assignment (s.d. = 25.1, 75% within 389 days, 90% within 399 days). The difference between experimental and control groups with respect to the length of time between random assignment and the followup interviews was not significant.

Tennessee. On average, the 117 initial interviews with caretakers were completed in 15.4 days (s.d. = 7.0 days) after random assignment. The length of time to completion for these interviews ranged from 2 days to 36 days (75% were completed in 20 days, 90% in 26 days). Similar to both the Kentucky and New Jersey caretaker interviews, these interviews were conducted, on average, two weeks into a four-week intervention. Therefore, the first interview should not be considered representative of the family's situation at the inception of family preservation or regular services. The 112 initial caseworker interviews were completed in an average of 14.5 days (s.d. 7.2 days) after random assignment, with a range of 3 to 34 days (75% completed within 18 days, 90% completed within 23 days). The length of time from random assignment to completion of the initial interview with caseworkers was significantly shorter for the control group than for the experimental group (11.8 days vs. 16.5 days, p = .0004).

The length of time between random assignment and the post-treatment interview was significantly different for experimental and control groups on both the caretaker and the caseworker interviews. Therefore, these timeframes are reported separately for each group. The 80 post-treatment interviews with caretakers in the experimental group were completed in an average of 47.9 days (s.d. = 10.3 days) after random assignment, while the 37 post-treatment interviews with control group caretakers were completed in an average of 51.9 days (s.d. = 10.6 days) after random assignment (p = .05). This time period ranged from 32 days to 80 days for the experimental group (75% in 54 days, 90% in 64 days) and from 29 days to 70 days for the control group (75% in 61 days, 90% in 67 days). For the caseworker interviews, the 90 post-treatment interviews in the experimental group were completed an average of 59.7 days (s.d. = 18.5 days) after random assignment, whereas the 48 post-treatment interviews in the control group were completed an average of 47.8 days (s.d. = 9.7 days) after random assignment (p = .0001). The length of time from random assignment to the initial caseworker interview ranged from 37 to 135 days (75% within 68 days, 90% within 83 days) for the experimental group, and from 36 days to 91 days (75% within 48 days and 90% within 61 days) for the control group.

One hundred and eight caretakers completed both the initial and post-treatment interviews, with an average of 33.6 days between the interviews (s.d. = 10.1 days, 75% with not more than 41 days between interviews, 90% with not more than 47 days). For the 112 caseworkers completing both the initial and post-treatment interviews, there was a significant difference between the experimental and control groups in length of time between interviews (p = .03). Forty-six caseworkers in the control group completed both interviews with an average of 35.7 days between the interviews (s.d. = 12.3, 75% with no more than 38 days between, 90% with no more than 55 days). Sixty-six caseworkers in the experimental group completed both interviews with an average of 42.3 days between the interviews (s.d. = 17.5, 75% with not more than 49 days between, 90% with not more than 60 days between).

Followup interviews were completed by 95 caretakers an average of 385.5 days after random assignment (s.d. = 25.9, 75% within 394 days, 90% within 404 days). The difference between experimental and control groups with respect to the length of time between random assignment and the followup interviews was not significant.

Philadelphia. A total of 263 initial interviews with caretakers were completed, and the length of time from random assignment to completion of these interviews ranged from 9 to 85 days with an average of 28.54 days (s.d. = 11.28 days, 75% within 36 days, 90% within 44 days). For the 163 caseworker initial interviews, the length of time from random assignment to interview completion ranged from 12 days to 86 days with an average time of 37.12 days (s.d. = 16.11 days, 75% within 45 days, 90% within 59 days). For both the caretaker and the caseworker initial interviews, the length of time between random assignment and interview completion was significantly longer for the control group. For caretaker initial interviews, the treatment group averaged 26.54 days (s.d. = 26.54 days) and the control group averaged 31.46 days (s.d. = 31.46 days, p = .0009). For the caseworker initial interviews, the average number of days between random assignment and interview completion in the treatment group was 34.07 days (s.d. = 14.49 days), and the average number of days in the control group was 44.0 days (s.d. = 17.57 days, p = .0002).

At the end of the treatment period, 261 caretakers were interviewed, and the length of time from random assignment to interview completion ranged from 68 days to 180 days with an average of 111.55 days (s.d. = 13.58 days, 75% within 118 days, 90% within 129 days). For the caseworkers, 250 post-treatment interviews were completed in an average of 129.48 days (s.d. = 28.43 days) after random assignment, with a completion time ranging from 100 days to 270 days (75% within 140 days, 90% within 164 days). There were no significant differences between experimental and control groups in the length of time from random assignment to post-treatment interview completion for either the caretaker or the caseworker interviews.

Two hundred and twenty-five followup interviews were completed with caretakers. The length of time from random assignment to the followup interview ranged from 223 days to 438 days with an average of 360.59 days (s.d. = 37.95, 75% within 379 days, 90% within 391 days). Experimental and control groups did not differ significantly in the amount of time between random assignment and followup caretaker interviews.

As noted, the first interview was to be conducted within two weeks of the referral and the second interview was to be conducted at the end of service provision or a comparable time. Therefore, it is expected that for those cases where both interviews were completed, approximately 12 weeks (or 90 days) should have passed between the dates that the first and second interviews were conducted. Two hundred and nineteen caretakers completed both the first and second interviews, and the average length of time between these interviews was 81.21 days (s.d. = 14.94 days, 75% with 75 days between interviews, 90% with 90 days between interviews). For the 160 caseworkers who completed both interviews, the average length of time between these interviews was 89.57 days (s.d. = 27.68 days, 75% with 103 days between interviews, 90% with 123 days between interviews). The average number of days between initial and post-treatment caretaker interviews was significantly greater for the experimental group than the control group (124 days vs. 95 days, p = .007). The same was true for the time between caseworker initial and post-treatment interviews, with an average of 112 days for the experimental group and 48 days for the control group (p = .0001).

[Go To Contents]

2.6 Administrative Data

We attempted to gather administrative data on substitute care placements and reports of maltreatment both before and after assignment into the study on all of the net study cases. This administrative data also contained other information such as case opening dates, types of maltreatment, and some demographic data. In Kentucky, of the 358 randomly assigned cases, no administrative data were obtained from DSS on 3 cases, an additional case (1) had no recent activity in the administrative data, ⁽²⁰⁾ and as already noted, 9 cases were inappropriate referrals. These 13 cases were excluded from the administrative data analyses. ⁽²¹⁾ In New Jersey, we obtained administrative data on all of the 442 net study cases (100%), 275 in the experimental group and 167 in the control group (Table 2-8). New Jersey administrative data included some information on services other than placement. In Tennessee, we obtained information on placement and reports of maltreatment from administrative data and case records. Placement data were available for 140 (95%) of the cases, 47 in the control group and 93 in the experimental group. Allegation data were available for 144 (98%) cases, 48 in the control group and 96 in the experimental group. In Philadelphia, administrative data were available for all but 4 of the 353 cases.

2.7 Maintaining Study Integrity

It was through the site coordinator activities that many aspects of the study integrity were controlled. This was accomplished in a variety of ways. The site coordinators served as the points of contact between the home office and agency liaisons. They monitored performance by the participating agencies, alerted the home office to problems, and became actively involved in resolving problems as they arose.

The site coordinator (SC) was responsible for tracking down needed information to complete interviews (e.g., addresses, caseworker names). Additionally, the SC monitored the status of individual cases to report changes in service end dates, or to identify and seek explanations for cases in which the assignment to regular or experimental services appeared to have been violated. These included cases that should have been but were not referred to random assignment, cases that were randomly assigned but did not get referred to the appropriate service provider, and cases that were not eligible for the study, but were receiving family preservation services. This was accomplished by comparing results of random assignment to agency logs on a monthly basis. State and local personnel were provided monthly reports delineating the cases assigned, their status, and problem areas.

The site coordinator also had a weekly meeting with the public agency screeners and private agency liaisons to review concerns and problems. By keeping in touch with caseworkers and persons in critical positions to the project, the SC was able to gather information about changes in policies, procedures, and staff so that necessary changes could be made. In Kentucky, Tennessee, and Philadelphia there was one site coordinator for one site, while in New Jersey, two site coordinators traveled across seven counties.

[Go To Contents]

Endnotes

10. The Family Preservation and Family Support Implementation Study was selecting sites at the same time. It was decided that conducting both studies in the same site would be too burdensome for states: therefore, Alabama, Arizona, Texas, and Los Angeles, California were eliminated as candidates for the second round of site visits.

11. The study was also conducted in Lexington for a limited period of time.

12. Kentucky state social services have since been reorganized. DSS merged with the Department for Social Insurance to become the Department for Community Based Services.

13. In both Kentucky and New Jersey, two families were randomly assigned twice. The second of these assignments was considered an inappropriate referral and was dropped from this count.

14. One control group case in Kentucky was classified as "minimal service" because the family moved to another state shortly after the referral.

15. Staff indicated there were no contacts for 25 cases in Kentucky (18E and 7C); 14 cases in New Jersey (5E and 9C), 7 cases in Tennessee (6C and 1E.), and 143 cases in Philadelphia (58C and 85E).

16. Transferring cases was not a problem for experimental cases as they went directly to a family preservation worker.

17. One case in the experimental group was a turnback where the second interview was conducted with the public agency worker and the first interview was conducted with the FPS worker 20 days after the second interview had already been conducted. For this case and four others where there were less than 10 days between the two caseworker interviews, computed scores measuring the change between initial and post-treatment interviews were dropped from the caseworker data.

18. The difference in times to interview for the caretaker post-treatment interviews was nearly significant: experimental group, 51.2 days vs. 53.7 days for the control group, p = 0.056.

19. Two cases in the experimental group were closed by the time the worker was contacted for the initial interview, so both caseworker interviews were conducted on the same day. In all, 9 sets of initial and post-treatment interviews (3 caretaker and 6 caseworker) were conducted with less than 10 days between completion dates. For these cases, computed scores measuring the change between initial and post-treatment were dropped from the caseworker data.

20. For all cases in Kentucky, we calculated the length of time between the last activity recorded in the administrative data before referral to family preservation services and the date of referral to family preservation services. For each of these 20 cases, there was no recorded activity within 3 years prior to the referral date. It appears that for these cases, recent administrative data were not obtained from the DSS system.

21. In the course of the evaluation, Kentucky changed administrative data systems, which resulted in some difficulties in the retrieval of administrative data.

Where to?

Top of Page | Contents

Main Page of Report | Contents of Report

Home Pages:
Human Services Policy
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)