Evaluation of Family Preservation and Reunification Programs: Final Report - Volume One. 2.7 Maintaining Study Integrity


It was through the site coordinator activities that many aspects of the study integrity were controlled. This was accomplished in a variety of ways. The site coordinators served as the points of contact between the home office and agency liaisons. They monitored performance by the participating agencies, alerted the home office to problems, and became actively involved in resolving problems as they arose.

The site coordinator (SC) was responsible for tracking down needed information to complete interviews (e.g., addresses, caseworker names). Additionally, the SC monitored the status of individual cases to report changes in service end dates, or to identify and seek explanations for cases in which the assignment to regular or experimental services appeared to have been violated. These included cases that should have been but were not referred to random assignment, cases that were randomly assigned but did not get referred to the appropriate service provider, and cases that were not eligible for the study, but were receiving family preservation services. This was accomplished by comparing results of random assignment to agency logs on a monthly basis. State and local personnel were provided monthly reports delineating the cases assigned, their status, and problem areas.

The site coordinator also had a weekly meeting with the public agency screeners and private agency liaisons to review concerns and problems. By keeping in touch with caseworkers and persons in critical positions to the project, the SC was able to gather information about changes in policies, procedures, and staff so that necessary changes could be made. In Kentucky, Tennessee, and Philadelphia there was one site coordinator for one site, while in New Jersey, two site coordinators traveled across seven counties.

Table 2-8
Numbers of Cases on which Administrative Data are Available by County
  Jefferson Fayette Total KY
Net study cases 162 155 13 19 175 174
Cases with administrative data 160 155 13 17 173 172
Note: Administrative data on one KY case in the experimental group contained only opening and closing data on an adult family member. No data on placements or reports of maltreatment were available for this case.
New Jersey
  Camden Burlington Ocean Monmouth Essex Bergen Passaic Total NJ
  C E C E C E C E C E C E C E C E
Net study cases 19 39 20 47 29 41 23 25 45 62 20 29 11 32 167 275
Cases with administrative data 19 39 20 47 29 41 23 25 44 62 20 29 11 32 167 275
Tennessee- Shelby Philadelphia
  C E   C E
Net study cases 49 98 Net study cases 144 209
Cases with administrative data 48 96 Cases with administrative data 144 205


10. The Family Preservation and Family Support Implementation Study was selecting sites at the same time. It was decided that conducting both studies in the same site would be too burdensome for states: therefore, Alabama, Arizona, Texas, and Los Angeles, California were eliminated as candidates for the second round of site visits.

11. The study was also conducted in Lexington for a limited period of time.

12. Kentucky state social services have since been reorganized. DSS merged with the Department for Social Insurance to become the Department for Community Based Services.

13. In both Kentucky and New Jersey, two families were randomly assigned twice. The second of these assignments was considered an inappropriate referral and was dropped from this count.

14. One control group case in Kentucky was classified as "minimal service" because the family moved to another state shortly after the referral.

15. Staff indicated there were no contacts for 25 cases in Kentucky (18E and 7C); 14 cases in New Jersey (5E and 9C), 7 cases in Tennessee (6C and 1E.), and 143 cases in Philadelphia (58C and 85E).

16. Transferring cases was not a problem for experimental cases as they went directly to a family preservation worker.

17. One case in the experimental group was a turnback where the second interview was conducted with the public agency worker and the first interview was conducted with the FPS worker 20 days after the second interview had already been conducted. For this case and four others where there were less than 10 days between the two caseworker interviews, computed scores measuring the change between initial and post-treatment interviews were dropped from the caseworker data.

18. The difference in times to interview for the caretaker post-treatment interviews was nearly significant: experimental group, 51.2 days vs. 53.7 days for the control group, p = 0.056.

19. Two cases in the experimental group were closed by the time the worker was contacted for the initial interview, so both caseworker interviews were conducted on the same day. In all, 9 sets of initial and post-treatment interviews (3 caretaker and 6 caseworker) were conducted with less than 10 days between completion dates. For these cases, computed scores measuring the change between initial and post-treatment were dropped from the caseworker data.

20. For all cases in Kentucky, we calculated the length of time between the last activity recorded in the administrative data before referral to family preservation services and the date of referral to family preservation services. For each of these 20 cases, there was no recorded activity within 3 years prior to the referral date. It appears that for these cases, recent administrative data were not obtained from the DSS system.

21. In the course of the evaluation, Kentucky changed administrative data systems, which resulted in some difficulties in the retrieval of administrative data.

View full report


"report1.pdf" (pdf, 770.39Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®