National Study of Child Protective Services Systems and Reform Efforts. Findings on Local CPS Practices. A.5 Weighting

05/01/2003

The weighting procedures included five steps — creating county-level base weights, creating agency-level weights, adjusting for nonresponse, creating replicate weights, and creating an agency-level analysis file. Each step is described in more detail below.

The 375 counties were selected with an equal probability within each of four strata defined by urbanicity (0 = rural, 1 = urban) and administration type (0 = state, 1 = county). The county-based weight was computed as the inverse of the county selection probability, that is, Nh/nh where Nh was the number of counties on the frame in stratum h; and nh was the number of sampled counties in stratum h. The county-based weights are shown in Table A-4.

Stratum Number of counties on frame Number of sampled counties County- based weight
Table A-4:
County-Based Weights
00 1,662 120 13.850
01 631 90 7.011
10 494 73 6.767
11 354 92 3.848
Total 3,141 375  

All 383 agencies that served the sampled 375 counties were included in the survey. Each agency was asked to report on all counties served besides the county from which the agency was sampled. While most agencies reported that they served one county, some agencies reported that they served multiple counties including unsampled counties. There were also some counties that were served by more than one agency. A more complicated situation occurred when a county was served by more than one agency, and one of the agencies serving that county also served other counties. (For Example, agency X located in sampled County A reported that it also served sampled County B, as well as nonsampled County C. Moreover, County B was served by another agency (Y), which reported that it also served sampled County A.)

To handle these various cases, a counting rule that could take care of these complexities was needed. The counting rule proposed by Sirken for network sampling was used.3 The rule assigns to each sampled agency a multiplicity factor that is the number of all the counties the agency served regardless of whether the counties were sampled or not. In the above example the multiplicity factor for agency X would be 3 and the multiplicity factor for agency Y would be 2. The distribution of the multiplicity factors is shown in Table A-5.

Multiplicity Number of agencies Number of counties served
Table A-5:
Distribution of Multiplicity Factors
1 285 285
2 40 80
3 27 81
4 5 20
5 9 45
6 8 48
7 8 56
8 1 8
Total 383 623

To assign weights to the agencies selected through the sampled counties, a county and agency combined file was created with a separate record for each unique sampled county and agency combination. Therefore, a sampled county appeared in the file as many times as it was linked to a sampled agency and the agency appeared in the file as many times as the number of sampled counties it served. In the above example, the combined file would contain four separate records as shown in Table A-6.

Agency ID County ID Multiplicity
Table A-6:
Example of Agency Records
in the Combined File
X05 086 3
X05 090 3
Y09 090 2
Y09 086 2

This file contained 425 records, of which each was assigned a multiplicity-adjusted weight established by dividing the county-level base weight by the multiplicity factor. Note that duplicate agency records were assigned their own weights, which could be different if the county-level base weights were different. These weights were adjusted for nonresponse as explained below.

Agency nonrespondents were those whose participation was declined by their States or by their counties and those who did not return their survey questionnaires (Table A-7).

Agency disposition Frequency
Table A-7:
Frequency Distribution
of the Agency Dispositions
1 (survey returned) 307
2 (state declined) 31
3 (county declined) 36
4 (survey not returned) 9
Total 383

The SPSS procedure called Chi-Square Automatic Interaction Detection (CHAID) was considered for determining the significant variables for predicting response; and the significant predictors would then be used to form weighting classes for nonresponse adjustment. The variables used as the predictors were county characteristics, including per capita income, urbanicity, administrative structure, and Census region. For agencies that served multiple counties, the predictor variables were based on the characteristics of the county with the largest population. The population data were obtained from the July 1, 2001, Census county population estimates. Table A-8 shows the distributions of response status of the sampled agencies by stratum. The overall response rate was the targeted 80 percent.

Stratum Respondents Nonrespondents Total Sample distribution Response rate
Table A-8:
Total Agency Sample by Response Status and Urbanicity/Administration Type
Rural/State-administered (00) 97 26 123 32% 79%
Rural/county-administered (01) 74 16 90 24% 82%
Urban/State-administered (10) 59 19 78 20% 76%
Urban/county-administered (11) 77 15 92 24% 84%
Total 307 76 383 100% 80%

The CHAID analysis indicated that none of the predictor variables were significant in predicting agency response. Thus, a single nonresponse adjustment factor could be used for the whole sample. However, the sampling strata were still used to define the nonresponse adjustment cells as it is simpler for variance estimation to confine all weighting adjustments within the sampling strata. Within a nonresponse adjustment cell, the nonresponse adjustment factor was computed as:

The sum of Multiplicity adjusted weights for all sampled agencies divided by the sum of Multiplicity adjusted weights for all responding agencies.

In the summations, all duplicate agency records were added separately. The nonresponse adjustment factors are shown in Table A-9.

Stratum (adjustment cell) Number of sampled agencies Number of responding agencies Nonresponse adjustment factor
Table A-9:
Nonresponse Adjustment Factors by Sampling Stratum
00 151 125 1.363
01 90 74 1.221
10 84 65 1.403
11 100 85 1.202
Total 425 349  

The final agency-level weights were then computed as the product of the multiplicity adjusted weights and the nonresponse adjustment factor. The sum of all agency-level final weights provided an estimate of the number of CPS agencies in the nation equal to 2,607 agencies.

The jackknife (JKn) method for variance estimation was used for analyses. This method was implemented using WesVar.4 WesVar uses the JKn method for variance estimation, which requires replicate weights. The JKn method is appropriate for stratified sampling with more than two variance units per stratum.

The replicates for the JKn method were created by deleting one variance unit at a time and adjusting the weights for other variance units from the same variance stratum but leaving the other weights unchanged. The sampling strata constituted the variance strata but the variance units were formed by randomly grouping sampled counties within each stratum. Then, WesVar was used to create the replicate weights.

The variance units (VarUnits) were created to have five counties in each variance unit within each variance stratum (VarStrat). In the JKn method, the number of replicates, G, was equal to The sum from h=1 to h=4 of m sub h. where mh was the number of variance units in variance stratum h. There were 74 replicates, one replicate corresponding to each VarUnit. The replicate weights were formed by deleting one VarUnit at a time and adjust the weights for counties in other VarUnits in the same VarStrat. The deletion of a VarUnit was equivalent to assigning zero weight to the counties in the deleted VarUnit. For example, the replicate weights for the first replicate are defined as follows:

  • Zero for counties in VarUnit 1 in VarStrat 1;
  • m1 divided by m1 minus 1. Times county-base weight for counties in other VarUnits 2-24 in VarStrat 1; and
  • County-based weight for all others in VarStrat 2-4 (i.e., no change in their weights).

The remaining 73 replicates and their replicate weights were formed in the same manner. The replicate creation is summarized in Table A-10.

Stratum VarStrat Number of counties VarUnits Number of replicates
Table A-10:
Summary of Replicate Creation
00 1 120 1 -24 24
01 2 90 1 -18 18
10 3 73 1 -14 14
11 4 92 1 -18 18
Total   375   74

Next, agency-level replicate weights were computed by dividing the county replicate weights by the multiplicity factor. Nonresponse adjusted agency level replicate weights were also computed.

The final step in the weighting process was to manipulate the weights to reconfigure the combined file into an agency-level file that contained only unique agency records. This was done by aggregating the weight fields at the agency level.