[ Main page of Report | Contents of Report ]
This appendix describes the methodology for the LAS, including information on sampling, instrument design, data collection, database development, weighting, and analysis.
The sampling procedures were designed to select a nationally representative sample of counties. The analysis unit was the local CPS agency, which in the majority of cases was operated at the county level. As a comprehensive listing of CPS agencies did not exist, the county was used as the primary sampling unit and a list of all U.S. counties was the sampling frame.
The sampling process incorporated two key features of the stratified systematic random sampling approach — the stratification structure used in the sampling process and differential sampling rates for different sectors of counties.(1) Stratification and differential sampling rates were designed to ensure adequate representation of counties by the different categories of CPS administrative structure (either State- or county-administered) and urbanicity (urban or rural).
The stratification variables were selected based on their assumed underlying association with CPS system characteristics. The administrative structure variable mediated the degree to which State policy affects local CPS agency operations, while the urbanicity variable addressed different operational environments and resources. In addition, a Census geographic region variable was used to sort the sampling list to ensure that the sample was spread evenly across regions. Table A-1 lists the States in each of the four census regions.
| Region | States |
|---|---|
| 1 | CT, MA, ME, NH, NJ, NY, PA, RI, VT |
| 2 | IA, IL, IN, KS, MI, MN, MO, ND, NE, OH, SD, WI |
| 3 | AL, AR, DC, DE, FL, GA, KY, LA, MD, MS, NC, OK, SC, TN, TX, VA, WV |
| 4 | AK, AZ, CA, CO, HI, ID, MT, NM, NV, OR, UT, WA, WY |
Information about the type of CPS administrative structure reflects the classification of State child welfare systems as those that are county-administered and those that are State-administered, as categorized by the American Public Human Services Association (APHSA). This State-level classification was attached to all the counties within the State. Below is a list of the States in which CPS services are county-administered.
The county urbanicity status was determined by the county's Metropolitan Statistical Area status (urban vs. rural) as classified by the Census Bureau. The following table provides the national distribution of counties by administrative structure and urbanicity (Table A-2).
| Administrative Structure | Urban | Rural | Total |
|---|---|---|---|
| County | 354 | 631 | 985 |
| State | 494 | 1,662 | 2,156 |
| Total | 848 | 2,293 | 3,141 |
The second key feature of the sampling process was the use of differential sampling rates. It was clear that the urban stratum was much smaller than the rural stratum, while the county-administered stratum was small compared to the State-administered stratum. The urban counties and the county-administered counties were oversampled (i.e. sampled at higher rates), which ensured that the data supported analyses within each stratum.
The sample allocation was carried out hierarchically. The total sample of 300 counties was allocated to the urban and rural strata by using sampling rates that differ by a ratio of 2:1. This means that the urban counties were selected with a sampling rate twice that of the rural counties. The sample was further allocated to the administrative structure strata by the same ratio of 2:1 in favor of the county-administered counties, which similarly oversampled those counties. The resulting sample target allocation is displayed in the first half of Table A-3. To achieve the targets in the final database, a larger sample was selected in order to allow for losses due to nonresponse or refusals to participate. In all, 375 counties were sampled. The fielded county sample sizes were calculated by assuming an expected response rate of 80 percent throughout all strata (bottom portion of Table A-3).
| Administration | Urban | Rural | Total |
|---|---|---|---|
| Target sample size | |||
| County | 74 | 72 | 146 |
| State | 58 | 96 | 154 |
| Total | 132 | 168 | 300 |
| Field sample size | |||
| County | 92 | 90 | 182 |
| State | 73 | 120 | 193 |
| Total | 165 | 210 | 375 |
With this sample size, it was estimated that the expected standard errors on estimates of national proportions would not exceed 3.4 percent, assuming a design effect of 1.35. This design effect was estimated with the formula, 1+C2, where C2 was the relative variance of the sampling weights of counties, and allowed 10 percent inflation to account for nonresponse adjustment and the difference between the sampling unit (county) and unit of analysis (agency).
A power analysis was also performed to ensure the sample size provided a reasonable power for statistical tests of subgroup comparisons. The power was calculated for a normal test of whether or not the proportions of the urban and rural strata were equal. Assuming that one proportion was 50 percent and the other was 65 percent, the expected power was 62 percent for the 5 percent significance level and 73 percent for the 10 percent significance level. This meant that a 15 percent difference in proportions would be detected with a similar level of probability (e.g., 62 percent probability if the significance level of the test was 5 percent).
A list of local CPS agencies was built, beginning with existing listings of these agencies, and verifying this information via phone calls to both State and local CPS authorities in the affected counties. All identified local CPS agencies that served the sample counties were recruited for data collection. Even in cases where one or more agencies served the sampled county (as well as other counties), all the agencies linked to the selected county were included in the agency sample. This resulted in 383 agencies being included in the survey. This sample design is a variant of the network sampling method, as sampled counties served as the network to identify local CPS agencies.(2)
[Go To Contents]
The first step in designing the LAS was to map the functions, activities, and components that encompass the work of CPS agencies into a list of topics. The topical list was organized by CPS function, but also included issues of policy, implementation, problems, and solutions. Once the topical list was developed, candidate questions were developed for review. The candidate questions in each topical area included some that had been tested and utilized in other surveys of CPS and other areas of child welfare. These questions were either adopted in their original form or adapted to meet the needs of this study.
The final data collection instrument was comprised of five modules. The first module inquired about the overall administration and organization of CPS. The next two modules focused on the two primary functions of CPS agencies — screening and investigation. The screening module included questions about the process by which the agency receives a referral concerning the welfare of a child and how the determination is made whether and how to respond to the referral. The investigation module included questions about the process by which the agency determines whether child maltreatment has occurred and/or the child is at risk of maltreatment.
The fourth module only applied to those agencies that have an alternative response option in addition to investigation. The fifth module included questions about the future directions of the CPS agency's administration and procedures. The agencies were instructed to have the most knowledgeable person complete each module.
Prior to finalizing the survey instrument, the LAS was pilot tested with nine agencies and reviewed by a number of experts in CPS practices. The comments and suggestions from the pilot test and expert review were incorporated into the final survey instrument.
[Go To Contents]
The survey was mailed to all 383 CPS agencies that were identified as serving the 375 sampled counties. Data collection lasted for 19 weeks — beginning February 1, 2002, and ending June 14, 2002. The targeted response rate was achieved — 307 sampled agencies completed the survey, for a response rate of 80 percent.
To facilitate data collection, 6 groups of approximately 60 to 65 agencies were targeted each week for data collection. Four main activities were involved in data collection — State solicitation, agency solicitation, survey distribution and retrieval, and nonresponse followup.
Prior to data collection, State child welfare directors were contacted to request permission to contact the sampled counties. Seven States declined to participate in the survey during this phase, which resulted in a loss of 63 potential responding agencies. Next, an initial letter was mailed to the director of each local CPS agency that introduced the study and requested the agency's participation. In addition to the letter, a "Frequently Asked Questions" sheet and a 5-page overview of the study were sent.
A confirmation call was made to the local CPS agency directors approximately 1 week after the solicitation letters were sent. The purpose of the call was to confirm the agency's participation and to respond to any questions the agency director might have. Generally, the survey was well received and most agencies agreed to consider participating. However, 36 agencies initially declined to participate in the survey.
During the confirmation calls, participating agencies were given preliminary instructions on how to complete the surveys. Multicounty agencies (CPS agencies that provide services to more than one county) and agencies that served multiagency counties (CPS agencies that share responsibility with one or more other CPS agencies for serving a single county) were given instructions on how to complete the survey to accommodate their special circumstances. Multicounty agencies that provided services to more than one county in the sample were instructed to complete only one survey for all the counties they served as long as the same practices and procedures were used. Agencies that used different procedures for different counties were instructed to complete a separate survey for each county. Agencies in multiagency counties were instructed to respond to the survey based solely on the section of the county they served.
Given the number of initial State and agency refusals, special efforts were necessary to boost the number of participating agencies to reach the targeted response rate of 80 percent. Senior members of the research team contacted the seven States that had initially denied permission for sampled counties to be included. This effort was successful and enlisted 3 additional States for the survey, which increased the number of potential respondents by 34 agencies.
An additional effort was made to enlist the support of the County Welfare Director's Association in one large State where, despite State CPS agency approval of the study, some agencies would not participate without the endorsement of the State's County Welfare Director's Association. Contact was made with the director of that association who then agreed to support the survey effort. This increased the number of potential respondents by another 13 agencies.
A receipt control database was developed to assist in administering the LAS, primarily to monitor survey progress and to create mailing lists. The tracking database recorded all contact information and agency participation status. The database was updated after each contact with an agency, as well as whenever a completed survey was received.
Survey packages were sent to those agencies that initially agreed to participate. Survey packages were also sent to any agency that had not been reached after 1 week of confirmation calls. The survey packages consisted of a confirmation letter, the five survey modules, and a return Federal Express envelope. Participating agencies were given 3 weeks to complete and return the survey.
Participating local agencies were given a toll-free telephone number and an email address to contact the survey administration team if they had any questions about the survey. The toll-free telephone number and email address were checked for incoming messages at least twice daily.
A number of efforts were taken to increase the response rate among those agencies that had initially agreed to participate in the survey but had not returned a survey. First, reminder postcards were mailed to agencies that had not returned their completed surveys within 2 weeks of receiving the initial survey package. A second survey package was sent to nonresponding agencies 4 weeks after the initial survey packages were sent. Followup calls were made to agencies that had not returned completed surveys 6 weeks after receiving their initial survey package. In addition, at the end of the survey administration period, State child welfare directors were requested to encourage nonresponding counties (and counties that had originally declined to participate in the survey) to respond.
[Go To Contents]
This section discusses the activities involved in developing the LAS database, including database design, quality control checks, resolution of database errors, and coding of text responses.
A database was designed for data entry with numeric fields to capture nominal or scale responses and text fields for text responses. Survey data were entered into the database exactly as recorded on the survey forms. The database file was saved in comma-delimited format and exported into the Statistical Package for the Social Sciences™ (SPSS) program, along with variable and value labels.
Several steps were taken to ensure the accuracy, validity, and internal consistency of data in the LAS database. A quality control check was performed on all returned surveys to verify that all modules were returned and complete. Problems encountered during this check were recorded on a quality control sheet, for followup with respondents prior to data entry.
Problems encountered during data entry (e.g., illegible responses, questions responded to incorrectly, etc.) were also documented on the quality control sheet. In addition, the database was designed to limit data entry errors by only accepting valid responses. Finally, to check for errors and make corrections, quality control and validity checks were designed and run in SPSS.
To facilitate the resolution of quality control errors in the database, the errors were divided into the following categories:
Different mechanisms were employed to resolve errors depending on their type. First, the team checked all errors by reviewing the hardcopy survey. When an error could not be resolved by a review of the questionnaire, an attempt was made to contact the survey respondent for clarification for the following types of errors — multiple responses, missing data, illegible responses, internal inconsistency, and out of range responses, as well as for missing modules and incomplete modules.
Skip pattern errors were recoded in SPSS. The recode was designed to accept the most logical correct response, and recode, or deselect, the "root" or followup response. In some cases, the skip pattern errors resulted in the creation of new codes. For example, some questions were frequently answered with a multiple response, when only one response was expected. Because of the frequency of this error, an additional code was developed to capture both responses selected. This allowed the data to more accurately reflect the actual practice at CPS agencies.
Finally, some errors were resolved through followup with State-level child welfare agencies. For example, after several counties in one State completed and returned their surveys, the State child welfare director indicated that on a few variables, the agencies had provided incorrect statistics. The director wanted to review the completed surveys for these counties, and provide the correct information, if necessary. The survey administration team provided the State with copies of the completed surveys, and the State in turn provided corrected surveys.
Codes were established for text response questions in some of the survey modules and added to the database. All of the open-ended responses in the fifth module were coded. In Modules 2, 3, and 4, the text responses describing the agency's responsibility for different forms of maltreatment were coded. Otherwise, the remaining text response and other/specify fields were coded only if there were more than 45 responses.
[Go To Contents]
The weighting procedures included five steps — creating county-level base weights, creating agency-level weights, adjusting for nonresponse, creating replicate weights, and creating an agency-level analysis file. Each step is described in more detail below.
The 375 counties were selected with an equal probability within each of four strata defined by urbanicity (0 = rural, 1 = urban) and administration type (0 = state, 1 = county). The county-based weight was computed as the inverse of the county selection probability, that is, Nh/nh where Nh was the number of counties on the frame in stratum h; and nh was the number of sampled counties in stratum h. The county-based weights are shown in Table A-4.
| Stratum | Number of counties on frame | Number of sampled counties | County- based weight |
|---|---|---|---|
| 00 | 1,662 | 120 | 13.850 |
| 01 | 631 | 90 | 7.011 |
| 10 | 494 | 73 | 6.767 |
| 11 | 354 | 92 | 3.848 |
| Total | 3,141 | 375 |
All 383 agencies that served the sampled 375 counties were included in the survey. Each agency was asked to report on all counties served besides the county from which the agency was sampled. While most agencies reported that they served one county, some agencies reported that they served multiple counties including unsampled counties. There were also some counties that were served by more than one agency. A more complicated situation occurred when a county was served by more than one agency, and one of the agencies serving that county also served other counties. (For Example, agency X located in sampled County A reported that it also served sampled County B, as well as nonsampled County C. Moreover, County B was served by another agency (Y), which reported that it also served sampled County A.)
To handle these various cases, a counting rule that could take care of these complexities was needed. The counting rule proposed by Sirken for network sampling was used.(3) The rule assigns to each sampled agency a multiplicity factor that is the number of all the counties the agency served regardless of whether the counties were sampled or not. In the above example the multiplicity factor for agency X would be 3 and the multiplicity factor for agency Y would be 2. The distribution of the multiplicity factors is shown in Table A-5.
| Multiplicity | Number of agencies | Number of counties served |
|---|---|---|
| 1 | 285 | 285 |
| 2 | 40 | 80 |
| 3 | 27 | 81 |
| 4 | 5 | 20 |
| 5 | 9 | 45 |
| 6 | 8 | 48 |
| 7 | 8 | 56 |
| 8 | 1 | 8 |
| Total | 383 | 623 |
To assign weights to the agencies selected through the sampled counties, a county and agency combined file was created with a separate record for each unique sampled county and agency combination. Therefore, a sampled county appeared in the file as many times as it was linked to a sampled agency and the agency appeared in the file as many times as the number of sampled counties it served. In the above example, the combined file would contain four separate records as shown in Table A-6.
| Agency ID | County ID | Multiplicity |
|---|---|---|
| X05 | 086 | 3 |
| X05 | 090 | 3 |
| Y09 | 090 | 2 |
| Y09 | 086 | 2 |
This file contained 425 records, of which each was assigned a multiplicity-adjusted weight established by dividing the county-level base weight by the multiplicity factor. Note that duplicate agency records were assigned their own weights, which could be different if the county-level base weights were different. These weights were adjusted for nonresponse as explained below.
Agency nonrespondents were those whose participation was declined by their States or by their counties and those who did not return their survey questionnaires (Table A-7).
| Agency disposition | Frequency |
|---|---|
| 1 (survey returned) | 307 |
| 2 (state declined) | 31 |
| 3 (county declined) | 36 |
| 4 (survey not returned) | 9 |
| Total | 383 |
The SPSS procedure called Chi-Square Automatic Interaction Detection (CHAID) was considered for determining the significant variables for predicting response; and the significant predictors would then be used to form weighting classes for nonresponse adjustment. The variables used as the predictors were county characteristics, including per capita income, urbanicity, administrative structure, and Census region. For agencies that served multiple counties, the predictor variables were based on the characteristics of the county with the largest population. The population data were obtained from the July 1, 2001, Census county population estimates. Table A-8 shows the distributions of response status of the sampled agencies by stratum. The overall response rate was the targeted 80 percent.
| Stratum | Respondents | Nonrespondents | Total | Sample distribution | Response rate |
|---|---|---|---|---|---|
| Rural/State-administered (00) | 97 | 26 | 123 | 32% | 79% |
| Rural/county-administered (01) | 74 | 16 | 90 | 24% | 82% |
| Urban/State-administered (10) | 59 | 19 | 78 | 20% | 76% |
| Urban/county-administered (11) | 77 | 15 | 92 | 24% | 84% |
| Total | 307 | 76 | 383 | 100% | 80% |
The CHAID analysis indicated that none of the predictor variables were significant in predicting agency response. Thus, a single nonresponse adjustment factor could be used for the whole sample. However, the sampling strata were still used to define the nonresponse adjustment cells as it is simpler for variance estimation to confine all weighting adjustments within the sampling strata. Within a nonresponse adjustment cell, the nonresponse adjustment factor was computed as:
In the summations, all duplicate agency records were added separately. The nonresponse adjustment factors are shown in Table A-9.
| Stratum (adjustment cell) | Number of sampled agencies | Number of responding agencies | Nonresponse adjustment factor |
|---|---|---|---|
| 00 | 151 | 125 | 1.363 |
| 01 | 90 | 74 | 1.221 |
| 10 | 84 | 65 | 1.403 |
| 11 | 100 | 85 | 1.202 |
| Total | 425 | 349 |
The final agency-level weights were then computed as the product of the multiplicity adjusted weights and the nonresponse adjustment factor. The sum of all agency-level final weights provided an estimate of the number of CPS agencies in the nation equal to 2,607 agencies.
The jackknife (JKn) method for variance estimation was used for analyses. This method was implemented using WesVar.(4) WesVar uses the JKn method for variance estimation, which requires replicate weights. The JKn method is appropriate for stratified sampling with more than two variance units per stratum.
The replicates for the JKn method were created by deleting one variance unit at a time and adjusting the weights for other variance units from the same variance stratum but leaving the other weights unchanged. The sampling strata constituted the variance strata but the variance units were formed by randomly grouping sampled counties within each stratum. Then, WesVar was used to create the replicate weights.
The variance units (VarUnits) were created to have five counties in each
variance unit within each variance stratum (VarStrat). In the JKn method,
the number of replicates, G, was equal to
where mh was the number of variance units in variance stratum
h. There were 74 replicates, one replicate corresponding to each
VarUnit. The replicate weights were formed by deleting one VarUnit at a time
and adjust the weights for counties in other VarUnits in the same VarStrat.
The deletion of a VarUnit was equivalent to assigning zero weight to the
counties in the deleted VarUnit. For example, the replicate weights for the
first replicate are defined as follows:
Times county-base weight for counties in other VarUnits 2-24 in VarStrat
1; and
The remaining 73 replicates and their replicate weights were formed in the same manner. The replicate creation is summarized in Table A-10.
| Stratum | VarStrat | Number of counties | VarUnits | Number of replicates |
|---|---|---|---|---|
| 00 | 1 | 120 | 1 -24 | 24 |
| 01 | 2 | 90 | 1 -18 | 18 |
| 10 | 3 | 73 | 1 -14 | 14 |
| 11 | 4 | 92 | 1 -18 | 18 |
| Total | 375 | 74 |
Next, agency-level replicate weights were computed by dividing the county replicate weights by the multiplicity factor. Nonresponse adjusted agency level replicate weights were also computed.
The final step in the weighting process was to manipulate the weights to reconfigure the combined file into an agency-level file that contained only unique agency records. This was done by aggregating the weight fields at the agency level.
[Go To Contents]
Planning for the analysis began prior to data collection with the preparation of a detailed outline of the analyses to be performed. For each chapter of the final report, the outline listed the topics and tables that would appear in the chapter. For each table, the outline showed the section of the report, the specific variables needed for analysis, the topic of the table, the unit of analysis for the table, an indication of whether derived or recoded variables were needed to perform the analysis, an indication of whether the table required a subset of the data, and the statistics to be included in the table. The outline also included table shells to show how the table would appear in the final report.
The next step in preparing for analysis was to derive the variables needed for analysis. Using the outlines described above, programmer specifications were written with instructions on how to create all of the needed variables. Once the SAS program had been written and executed, the output was reviewed to check the accuracy of the derivations.
Before beginning the analysis, several sets of variables were merged onto the main database. First, the derived variables and the full sample and replicate weights were merged. Then, a select group of variables from the State policy analysis database were added to the analytic file. This included variables classifying the county as State administered, county administered or State administered with strong county structure, based upon the policy review analyses. Once all of the needed variables were merged onto the database, the analysis file was uploaded into WesVar.
In designing the analyses, the project team considered that specific subgroups of agencies could potentially provide quite different response profiles on the survey. A number of agency characteristics were identified as potentially important subgroup markers in this regard. The first analytic task was to test the utility of these candidate categorization schemes in relation to answers on a select set of survey items. The candidate schemes identified by the project team included number of CPS staff, average referrals per CPS worker, overall level of agency responsibility for functions, number of disposition categories, number of response types, administrative structure, metropolitan status, urbanicity, and median income. The project team conducted analyses on certain key tables to identify which scheme would be most informative. The tables that were examined included the following:
The categorizations schemes were tested by producing the crosstabulations necessary to complete the key tables. The project team reviewed these findings and found that income and administrative structure emerged as good candidates for further exploration. Chi-square tests were performed for most of the key tables to determine whether the distributions were significantly different for the various income categories and for the various administrative structure categories (see Table A-11). County per capita income was used for the income analysis. The agencies were divided into two groups — above the median and below the median. The chi-square analyses compared the two groups for each of the key items with the results in the table. The agency administrative structure classification used information from the State policy reviews to classify each agency into one of three groups — State-administered, county-administered, or State-administered with strong county structure. For each key item, the Chi-square analyses compared these three groups.
| Key Table* | Income | Structure | ||
|---|---|---|---|---|
| X2 Value | X2 Probability | X2 Value | X2 Probability | |
| Overall responsibility for Screening/intake | 5.893 | .253 | 22.729 | .012 |
| New referral alleging maltreatment | 20.195 | .000 | 7.633 | .470 |
| Referral for child/household with an open investigation | 6.196 | .123 | 7.577 | .476 |
| Referral for child/household with prior substantiated report | 18.142 | .001 | 7.389 | .495 |
| Referral for child/household with prior unsubstantiated report | 16.041 | .003 | 7.39 | .495 |
| Referral for child who is in foster or substitute care | 1.834 | .676 | 7.458 | .488 |
| Type of response | 1.829 | .401 | 4.62 | .329 |
| Overall responsibility for investigation response | 6.154 | .188 | 32.054 | .000 |
| Overall responsibility for alternative response | 2.238 | .815 | 11.137 | .347 |
| Scope of investigation response | 1.962 | .521 | 15.278 | .018 |
| Scope of alternative response | 1.664 | .609 | 3.631 | .534 |
| Length of time current process in place: screening/intake | 0.487 | .912 | 3.203 | .619 |
| Length of time current process in place: investigation | 2.716 | .444 | 24.414 | .000 |
| Length of time current process in place: alternative response | 2.592 | .484 | 15.772 | .009 |
| Number of changes in last 6 months | 4.26 | .284 | 5.738 | .432 |
| * Includes those key tables with mutually exclusive categories. | ||||
The objective of this report is to present estimates of the percentage and number of the Nation's 2,610 CPS agencies . The data in this report are presented in one of several formats. When the rows of a table are mutually exclusive and total to 100 percent, the estimate, confidence interval, and percentage are given for each row. The 95-percent confidence interval provides a lower and upper bound for the estimate. This means that if the current study were exactly replicated 100 times, 95 of the replications would produce an estimate within this range. For other tables, the rows of the table are not independent and thus an agency can appear in any applicable row. For these tables, estimates and/or percentages based on the total number of agencies are given. For all tables, the estimates were rounded to the nearest 10.
[Go To Contents]
1. Sirken, M. G. (1972). Stratified Sample Surveys with Multiplicity. Journal of American Statistical Association, 67, 224-227.
2. Sirken, M. G. (1972). Stratified Sample Surveys with Multiplicity. Journal of American Statistical Association, 67, 224-227.
3. Sirken, M. G. (1972). Stratified Sample Surveys with Multiplicity. Journal of American Statistical Association, 67, 224-227.
4. Westat (2000). WesVar: WESVAR 4.0 User's Guide. Rockville, MD: Westat.
Main Page of Report | Contents of Report
Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)