National Study of Child Protective Services Systems and Reform Efforts. Findings on Local CPS Practices. A.1 Sampling Procedures


The sampling procedures were designed to select a nationally representative sample of counties. The analysis unit was the local CPS agency, which in the majority of cases was operated at the county level. As a comprehensive listing of CPS agencies did not exist, the county was used as the primary sampling unit and a list of all U.S. counties was the sampling frame.

The sampling process incorporated two key features of the stratified systematic random sampling approach — the stratification structure used in the sampling process and differential sampling rates for different sectors of counties.1 Stratification and differential sampling rates were designed to ensure adequate representation of counties by the different categories of CPS administrative structure (either State- or county-administered) and urbanicity (urban or rural).

The stratification variables were selected based on their assumed underlying association with CPS system characteristics. The administrative structure variable mediated the degree to which State policy affects local CPS agency operations, while the urbanicity variable addressed different operational environments and resources. In addition, a Census geographic region variable was used to sort the sampling list to ensure that the sample was spread evenly across regions. Table A-1 lists the States in each of the four census regions.

Region States
Table A-1:
Census Regions
1 CT, MA, ME, NH, NJ, NY, PA, RI, VT
2 IA, IL, IN, KS, MI, MN, MO, ND, NE, OH, SD, WI
3 AL, AR, DC, DE, FL, GA, KY, LA, MD, MS, NC, OK, SC, TN, TX, VA, WV
4 AK, AZ, CA, CO, HI, ID, MT, NM, NV, OR, UT, WA, WY

Information about the type of CPS administrative structure reflects the classification of State child welfare systems as those that are county-administered and those that are State-administered, as categorized by the American Public Human Services Association (APHSA). This State-level classification was attached to all the counties within the State. Below is a list of the States in which CPS services are county-administered.

  • California
  • Colorado
  • Georgia
  • Maryland
  • Minnesota
  • Nevada
  • New York
  • North Carolina
  • North Dakota
  • Ohio
  • Pennsylvania
  • Virginia
  • Wisconsin

The county urbanicity status was determined by the county's Metropolitan Statistical Area status (urban vs. rural) as classified by the Census Bureau. The following table provides the national distribution of counties by administrative structure and urbanicity (Table A-2).

Administrative Structure Urban Rural Total
Table A-2:
National Distribution of Counties
County 354 631 985
State 494 1,662 2,156
Total 848 2,293 3,141

The second key feature of the sampling process was the use of differential sampling rates. It was clear that the urban stratum was much smaller than the rural stratum, while the county-administered stratum was small compared to the State-administered stratum. The urban counties and the county-administered counties were oversampled (i.e. sampled at higher rates), which ensured that the data supported analyses within each stratum.

The sample allocation was carried out hierarchically. The total sample of 300 counties was allocated to the urban and rural strata by using sampling rates that differ by a ratio of 2:1. This means that the urban counties were selected with a sampling rate twice that of the rural counties. The sample was further allocated to the administrative structure strata by the same ratio of 2:1 in favor of the county-administered counties, which similarly oversampled those counties. The resulting sample target allocation is displayed in the first half of Table A-3. To achieve the targets in the final database, a larger sample was selected in order to allow for losses due to nonresponse or refusals to participate. In all, 375 counties were sampled. The fielded county sample sizes were calculated by assuming an expected response rate of 80 percent throughout all strata (bottom portion of Table A-3).

Administration Urban Rural Total
Table A-3:
Target and Field Sample Sizes
Target sample size
County 74 72 146
State 58 96 154
Total 132 168 300
Field sample size
County 92 90 182
State 73 120 193
Total 165 210 375

With this sample size, it was estimated that the expected standard errors on estimates of national proportions would not exceed 3.4 percent, assuming a design effect of 1.35. This design effect was estimated with the formula, 1+C2, where C2 was the relative variance of the sampling weights of counties, and allowed 10 percent inflation to account for nonresponse adjustment and the difference between the sampling unit (county) and unit of analysis (agency).

A power analysis was also performed to ensure the sample size provided a reasonable power for statistical tests of subgroup comparisons. The power was calculated for a normal test of whether or not the proportions of the urban and rural strata were equal. Assuming that one proportion was 50 percent and the other was 65 percent, the expected power was 62 percent for the 5 percent significance level and 73 percent for the 10 percent significance level. This meant that a 15 percent difference in proportions would be detected with a similar level of probability (e.g., 62 percent probability if the significance level of the test was 5 percent).

A list of local CPS agencies was built, beginning with existing listings of these agencies, and verifying this information via phone calls to both State and local CPS authorities in the affected counties. All identified local CPS agencies that served the sample counties were recruited for data collection. Even in cases where one or more agencies served the sampled county (as well as other counties), all the agencies linked to the selected county were included in the agency sample. This resulted in 383 agencies being included in the survey. This sample design is a variant of the network sampling method, as sampled counties served as the network to identify local CPS agencies.2