Deriving State-Level Estimates from Three National Surveys: A Statistical Assessment and State Tabulations. A. Supplement the State Samples


We identified three methods of increasing state sample sizes. Each requires advanced planning and additional funding, but, if designed correctly, would permit direct estimation for states with insufficient samples in the current surveys. States could be offered the opportunity to pay the incremental cost of additional data collection in their state.

1. Increase Existing Samples

The most straightforward procedure is to increase the sample sizes for existing surveys in the states, which currently have insufficient sample sizes. (Given the increased interest resulting from devolution in measuring outcomes at the state level, ASPE could also encourage the sponsoring agencies to put more emphasis on state sample sizes in future survey redesigns.)

These additional interviews would most likely be collected from the same primary sampling units (PSUs) currently used by the national survey. This avoids the significant additional costs associated with listing and interviewing in new PSUs. The Bureau of the Census (which conducts the data collection for these three surveys) may place additional restrictions on the within-PSU locations of the supplemental interviews in order to respect their complex rules that attempt to minimize the chance of respondents being interviewed for multiple Census surveys. For most states, this means that the data would continue to be collected from only a few parts of the state. For characteristics that vary significantly from one part of the state to another, point estimates, and their estimated precision, will both be subject to considerable variability. For example, in California, the proportion of the population that is Hispanic drops significantly as one moves from the southern to northern parts of the state. If states were willing to absorb the additional costs, it might be possible to work out an arrangement with the Bureau of the Census whereby additional PSUs could be included in the sample to improve the precision of the estimates.

2. Dual Frame Approach

Alternatively, state sample sizes can be increased using a dual-frame approach. Such a procedure would most likely involve a separate telephone survey that would ask (hopefully) identical questions as those posed in the in-person surveys. (Mail surveys of the general population tend to have response rates that are too low for Government surveys.) These telephone interviews would not be restricted to the PSUs of the in-person survey, rather they could be spread across the entire state. Sirken and Marker (1993) examined the possibility of combining the NHIS with state-level telephone surveys of various sizes. The telephone supplement would obviously be restricted to households with telephones. While only 5 percent of the United States population live in houses without telephones, the rate can be above 15 percent in some states. Many characteristics of interest to the Government are very different for households with and without telephones (Thornberry and Massey, 1986). In such cases, a decision would have to be made as to whether to use an unbiased estimator that weights households separately by whether or not they have a telephone, or a biased estimator with smaller variance that disregards this factor.

3. Add Questions to National Immunization Survey

A third procedure is to add questions to other existing Government surveys, for example, the National Immunization Survey (NIS). The NIS is a telephone survey that screens 900,000 households per year across the entire country to locate children aged 19-35 months and find out about their immunization rates (Ezzati-Rice et al., 1995). If the need for state estimates can be satisfied by the addition of a few questions, it may be economical to try and add them to the NIS screener. The incremental cost of including the few questions is likely to be much less than any attempt to conduct separate supplementary surveys. However, just like with the other dual-frame approaches described above, the extra households obtained from the NIS would be limited to those with telephones, requiring special weighting to retain unbiased estimates. Also, if many additional questions need to be asked, it is unlikely that the NIS or any other survey would be willing to include them in its questionnaire. The National Center for Health Statistics (NCHS) has conducted a pilot test for a State and Local Area Integrated Telephone Survey (SLAITS). SLAITS collects additional information from households screened for the NIS to produce state or local estimates. This June, NCHS plans to conduct another pilot SLAITS that will collect information of interest to ASPE for two states.

In general, the three surveys examined in this report have made great efforts to minimize nonsampling errors (e.g., Jabine et al., 1990). The questions used by the Bureau of the Census have been carefully pretested and the interviewers received detailed training. Given the common procedures used throughout the country for these surveys, nonsampling errors that do exist are likely to be similar across all states, rather than concentrated in a few. States with few PSUs, however, may be subject to interviewer effects because usually a single interviewer is assigned to a PSU. Nonsampling errors may be a more important issue when comparing, or combining, estimates from these three surveys with other surveys conducted by individual states subject to a distinct set of quality controls.

View full report


"deriving.pdf" (pdf, 2.83Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®