Preparation and training for the experiment were conducted in the summer of 1996. Training sessions were held with both DCS screeners and family preservation program coordinators. During the training sessions study procedures were reviewed including use of study forms, the screening protocol, random assignment procedures, and the role of the study site coordinator. A meeting was also held with all HomeTies staff and DCS staff. The purpose of the meeting was to train staff on Shelby procedures and targeting appropriate cases for family preservation. In addition, periodic group meetings were conducted with caseworkers and supervisors to reinforce study procedures and solicit their feedback on the study. Workers were very resistant to random assignment and concerned that it would deny services to families, cause extra tasks, and delay referrals for their caseloads.
A site coordinator was hired locally to assist DCS and HomeTies staff in providing case information needed for random assignment and the conduct of interviews.
Referral to Family Preservation. Prior to implementing study procedures, workers identified families they felt were appropriate to receive family preservation services, got supervisor approval for the referral, and then made the referral directly to the HomeTies program. The HomeTies worker would determine whether or not the referral was appropriate and if there were any openings. The evaluation slightly altered these procedures. The major change was that workers no longer directly referred cases to HomeTies. Instead, if a worker saw the need for in-home services, he or she asked the family to sign a release form to participate in the study. If the worker selected HomeTies, the worker then completed a random assignment form and a DCS screener checked that the primary child was under 13, not a juvenile court case, and not already in foster care. Screeners assessed whether the child was at imminent risk; it was up to the worker's supervisor to assess level of need.
The screener's role was to call HomeTies to see if there was an opening and if there was, contact Westat to randomly assign the case to the experimental or control group.
Cases eligible for the evaluation were limited to those served by child welfare, even though HomeTies also served cases referred from juvenile justice and mental health. Also, referrals were limited to those cases that were in the course of an investigation of abuse or neglect or shortly thereafter, and had at least one child under the age of 13 years old. A 60/40 (60% treatment) random assignment ratio was used at the beginning. This was changed to 70/30 when the evaluation was under way.
Impact of the study on DCS workers. Both CPS workers and supervisors expressed frustration about the impact of the study. Supervisors reported that there was no change in the characteristics of families referred to HomeTies after the study began. The most commonly cited problems resulting from the study included:
- "Emergencies happen around the clock"; front line staff could no longer make referrals directly to HomeTies after 4:30PM weekdays or during the weekends due to the office hours of DCS screeners. (Frayser staff estimate this accounted for 5-10 percent of their referrals.)
- With caseloads higher than usual due to the hiring freeze, workers did not have time "to scramble" to identify other services when the screener told them their cases were in the control group.
- Additional time related to the random assignment process. For families without telephones, workers had to make one trip to investigate and get consent to participate in HomeTies and then a second trip to tell families about the services they were to receive. To compound this, workers were still required to complete all of the paper work associated with a control group case.
- A worker could no longer choose a particular HomeTies worker to help with a case. This was troubling to some CPS workers who had built relationships and felt particularly confident in the abilities of certain workers.
- The "transition" was already requiring workers to review all of their open cases and close as many as possible. Workers and supervisors reported that they already had a lot of paperwork to complete and that the evaluation contact sheets were a very low priority.
- Workers and supervisors also reported getting calls from Westat about cases two months after they had closed the cases. By this time, workers did not remember all of the specifics and did not have time to go back through their records. If a worker left or was transferred, the supervisor had to answer the questions and it was even more difficult for them to know the specifics of a case.
In response to these issues, many workers reported that they were referring their families to other services instead of HomeTies to avoid "the hassle" of possible control group selection. They identified nine such programs. Some workers even asked staff within the Juvenile Court system to make referrals directly to HomeTies to ensure that cases got HomeTies services. Screeners estimated that only 20 percent of front line staff made referrals to HomeTies during the evaluation, whereas in the past, closer to 50 percent made referrals.
In response to the staff's "rebellion" against the study, administrators and CPS supervisors actively encouraged front line staff to use HomeTies. Due to the strain of the uncertainty of random assignment, supervisors reported that front line staff perceived the evaluation to be more cumbersome than it really was. The screeners reported that the local Westat site coordinator/data collector was "very motivating" to staff. Using information from case records, the data collector filled in gaps in the initial referral forms and completed contact forms for workers. Workers also had the option of filling out the contact forms instead of completing the case narrative in the case record.
Reduced referrals and financial issues. At the end of April, 1997, HomeTies was down 56 cases in comparison to budget projections (at approximately $2,500 per case, this is over $125,000), the program's worst financial year to date. Referrals were low before random assignment, and were reduced further after random assignment. The HomeTies program director was working with DCS to increase referrals. There was considerable frustration and hostility among some of the staff regarding random assignment and the reduction in referrals. While acknowledging that random assignment was not the only problem, one supervisor believed that promises had not been kept, stating that Westat had said that referrals would increase and the state had said that they would not allow HomeTies to suffer financially--neither of these things was happening. This person noted that people's livelihoods were in jeopardy and this had a big impact on worker's attitudes and on data collection.
During study interviews with staff, the program director of HomeTies said that low referrals were having negative financial implications on the program. The state was considering reimbursing the Frayser Family Counseling at a higher rate for HomeTies referrals for the rest of the year to make up for the shortfall. (Because of the lower number of staff, it was not clear the extent to which Frayser Family Counseling was actually losing money.) The program director stressed that the agency was not accepting different cases just to meet the budget, that is, clinical decisions were not to be affected by the present shortfall.
With regard to the assumption made prior to the decision to use random assignment in Tennessee - that more families needed services than were actually referred, one of the supervisors noted that the number of families in need of HomeTies had little to do with referrals to HomeTies. He noted that relatively few DCS workers actually referred to HomeTies and that there never had been enough referrals--it was very rare that HomeTies was not able to see a family within seven days.
To allay some of staff concerns, the random assignment was changed to 70 percent treatment and 30 percent control.
Other issues related to the research and its effects on practitioners. HomeTies supervisors identified a number of other concerns related to the research:
- Supervisors had to do much more hand holding with staff in supervision because of the stress of fewer referrals/lowered caseloads and the increased paperwork related to the research cases.
- Workers said that Westat forms asked them to be judgmental and blaming toward families.
- The yes-no questions were often impossible to answer, and didn't fit complex situations.
- Westat interviewers were often not available.
- Therapists didn't like doing the contact form for each visit and thought a weekly form could be used.
Therapists noted that clients said that they liked the gift certificate from McDonalds that they get for participating in interviews and that the Westat interviewers were nice. Only one family had an issue with the consent form.