The TEP weighed the strengths and limitations of four data sources for monitoring consumer perceptions of access including the Behavioral Risk Factor Surveillance System (BRFSS), the National Health Interview Survey (NHIS), the National Survey on Drug Use and Health (NSDUH), and the household component of the Medical Expenditure Panel Survey (MEPS). Additional detail on these data sources is included in Appendix B. Two datasets emerged as the best choices given HHS's needs. The BRFSS and NHIS could each be part of a system used by HHS to monitor Medicaid access at the state-level.
The BRFSS tracks health-related behaviors - including health risk behaviors, preventive health practices, and health care access - through a telephone survey administered in the United States each year. The key advantage of the BRFSS is that its large sample size of adults ages eighteen and older provides state-level estimates for all fifty states on an annual basis for several survey items on access to care. Its central limitations are that it does not currently ask about health insurance type (only whether someone is insured or uninsured) and it does not include data on access to care for children less than eighteen years of age. However, in 2013, BRFSS will include a question on type of health care insurance in its core questionnaire that will permit estimates of access to care for Medicaid beneficiaries at the state-level. The inclusion of the coverage type question in the core questionnaire beyond 2013 is pending funding. Continued funding to include the question on health insurance type in BRFSS is critical to track Medicaid beneficiaries' access to care (and relative access among those privately insured) and to support pre- and post-ACA analyses at the state-level. Also in 2013, a new BRFSS module on health care access will be added (see Appendix C). Currently forty-three states are planning to include this module.
Other limitations of BRFSS include concerns over state-level variation in administration and sampling design - particularly coverage of cell-phone-only households - as well as response rates, which are low in some states relative to other federally-sponsored surveys. However, design modifications made in 2011 included the use of cell phone interviewing and the proportion of interviews conducted with cell-phone-only households is expected to grow over time. Another limitation of BRFSS relates to the categorical income measure included in the survey, which yields less precise income data than other surveys. Some TEP members expressed concerns that the categorical income measure may not provide a precise, consistent measure of income or family income relative to poverty.
CDC's National Center for Health Statistics (NCHS) conducts the NHIS to capture information on a wide spectrum of health topics through personal household interviews. The major advantages of the NHIS are its inclusion of persons of all ages, multiple access-related items, a Medicaid-specific insurance variable, more detailed income data than the BRFSS, and a high response rate. Another advantage is that it is relatively easier to add questions to the NHIS than to BRFSS, because the former does not require approval from the states. The NHIS asks about all three types of care - preventive, primary, and specialty - and includes questions on wait times, transportation, and other financial and non-financial barriers to getting care, including mental health care.
Relevant to the inclusion of the NHIS as a data source for consumer perceptions of access is the NHIS sample augmentation and enhanced health care access and utilization questions that began in 2011. The sample augmentation permits additional estimates to be generated at the state-level and includes questions that address some of the provisions of the ACA. The NHIS sample size was increased in thirty-two states and the District of Columbia to increase the number of reliable state-level estimates that can be made on an annual basis. The largest states were not included in the sample augmentation because it was possible to produce state-level estimates with the usual NHIS sample. Using funds from the ACA, eighty-six additional questions on health care access and utilization were included on the NHIS for 2011-2013 in order to correspond with provisions of the ACA and allow for any early effects of the ACA to be assessed.
The primary limitation of the NHIS is the sample size and, consequently, whether the NHIS data can be used to generate state-level estimates of access for those enrolled in Medicaid. There are two distinct sampling issues that help inform the answer to this question. The first issue is whether the survey's sample is designed to produce unbiased estimates of the state's population (i.e., whether it can be used to produce state-representative data). The second issue is whether the effective sample size (i.e., the size of the sample after it is adjusted for complex survey design and weighting) for a specific measure produces a sufficiently precise estimate for the policy purposes needed.
The NHIS's current sample design allows for it to meet the design criteria. In 2011, the NHIS produced state-level insurance estimates for thirty-two states. In 2016 the NHIS is looking to change its sample design to produce representative estimates for even more states. The second issue of adequate precision is much harder to answer. An estimate of 15% of a population experiencing access problems in State X based on 100 effective sample size cases would have a standard error of 3.5% and a 95% confidence interval of 15%, plus or minus 7%. (An estimate of 15% based off of 50 effective sample cases would have a standard error of 5%, the same estimate with 200 effective sample size cases would have a standard error of 2.5%, and the same estimate with 400 effective sample size cases would have a standard error of 1.8%.)
Because the level of precision needed for an estimate to be policy relevant is dependent on how the estimate will be used, the decision of where to draw the line is not fixed. In making a judgment about how many cases per state are needed to achieve a level of precision for an estimate to be useful for HHS's policy assessments, NORC suggests a minimum of 100 effective sample size cases in the denominator of an estimate. This will mean that for most measures the actual number of cases needed in the state will be greater than 100. So to use the estimates from the NHIS for monitoring access to care among Medicaid enrollees, NORC recommends using only states that currently have adequate NHIS sample to be state representative if the number of Medicaid beneficiaries for any given measures is over 100 in the denominator of the estimate. A potential solution that would allow estimates to be generated for all fifty states is to use two- or three-year averages. Access to the NCHS-restricted data is required to conduct the necessary analyses to establish which states meet the effective sample size criteria (and a state that qualifies for one estimate on the list may not qualify for all estimates on the list). This access to the restricted data, including state identifiers, requires an application to the NCHS Research Data Center (RDC). Although minimal analyses can be conducted remotely, most analyses must be done onsite at an RDC. Because information on state sample sizes and population subgroups cannot be obtained without going to the RDC, determining whether these analyses are feasible has costs.
Two additional surveys were presented to the TEP for consideration as sources of indicators to measure consumer perceptions of access to care: the household component of MEPS and the NSDUH. Lack of an adequate sample size to generate state-level estimates and long lag times for release of data files limited any further consideration of MEPS. NSDUH was considered for its ability to capture subsets of Medicaid enrollees with mental illness and/or substance abuse issues, and for its questions specific to mental health and substance abuse. However, the relatively small number of Medicaid beneficiaries included at the state-level limits the extent to which NSDUH can be utilized for subgroup analyses and for generating estimates for low-prevalence conditions. Additionally, there is no known formal process for access to restricted data to attain state-level identifiers for NSDUH respondents.