Understanding Estimates of the Uninsured: Putting the Differences in Context

Understanding Estimates of the Uninsured: Putting the Differences in Context

ACKNOWLEDGMENTS: ASPE would like to thank those reviewers at the U.S. Census Bureau, Agency for Healthcare Research and Quality (AHRQ), the National Center for Health Statistics (NCHS), the Urban Institute, and the Center for Studying Health System Change, whose helpful comments and expertise contributed greatly to this document.

Introduction

The number of uninsured individuals in the United States has been an important policy concern for several years. The four federal surveys that are major sources of data on the uninsured have played an important role in informing this policy debate. These surveys – the Current Population Survey (CPS), the Medical Expenditure Panel Survey (MEPS), the National Health Interview Survey (NHIS), and the Survey of Income and Program Participation (SIPP) – can each provide useful estimates of the number of uninsured individuals during a particular period of time, and in some cases, at a point in time. Both MEPS and SIPP also provide information on changes in the insurance status of individuals over time. Two additional private sector surveys, the National Survey of America’s Families (NSAF) and the Community Tracking Study (CTS), also provide uninsured estimates. This note will explain some of the major reasons why estimates of the uninsured from these surveys differ and will explore various aspects of each survey.

General Survey Descriptions

• CPS: Conducted by the U.S. Census Bureau, the CPS is a monthly survey designed to collect data on the population’s employment status. The March supplement to the CPS asks additional questions about health insurance coverage. The CPS data are widely used because the CPS has the largest household survey sample size, is designed to produce credible state-level estimates (less populous states may require 3 years of CPS data to produce such estimates), is available on a timely basis, and provides information on coverage rates for socio-demographic subgroups of the population. However, as with most surveys, the CPS has been subject to questions regarding over- and under-reporting. Specifically, it is thought that the CPS over-counts the number of individuals who have been uninsured for an entire year, possibly because respondents answer based on current, rather than previous, coverage status. In addition, Medicaid coverage status is likely under-reported.¹
• MEPS: Administered by the Agency for Healthcare Research and Quality (AHRQ), MEPS is a two-year panel survey that collects information about health care use and expenditures, and the dynamics of health insurance and delivery systems. MEPS data on an individual’s health insurance status are collected several times a year and have the capacity to produce a variety of estimates of health insurance coverage, including point-in-time, monthly, and annual estimates. As a longitudinal survey, MEPS is able to examine health insurance dynamics, including changes in coverage and spells without coverage. MEPS data can be used to analyze the relationship between insurance status and a variety of individual and household characteristics, including use of and expenditures for health care services. Data on sources of payment for health care are collected and can be used to add greater accuracy and precision to the insurance status estimates. The MEPS sample is smaller than the CPS and NHIS samples, does not contain a representative sample from each state, and thus cannot be used to make state-level estimates.
• NHIS: Administered by the National Center for Health Statistics (NCHS), the NHIS is a continuous household interview survey of the civilian noninstitutionalized population. The NHIS collects detailed information on insurance status, type of coverage, measures of health status, health care utilization, and socio-demographic characteristics of members of sampled households. It provides estimates of the number of persons uninsured at a point-in-time, uninsured for a full-year, and ever uninsured during the year. The NHIS also collects data on illnesses, injuries, activity limitations, chronic conditions, health behaviors, and other health topics, which can be linked to characteristics such as insurance status. Like MEPS and SIPP, the NHIS does not contain a representative sample from each state. However, the NHIS design permits state level estimates in large states.
• SIPP: Also administered by the Census Bureau, SIPP is a 36-month longitudinal panel survey that collects information on health insurance status, income, labor force activity and federal program participation. As a longitudinal survey, SIPP data provide the capacity to examine the dynamics of health insurance. It measures the duration of spells without health insurance and provides health insurance estimates for various time periods, such as point-in-time, monthly, annual, or over the full panel. As part of the core data collected in the SIPP, health insurance data can also be linked to other sections of the survey, such as utilization of health care services, child well-being, and disability. The SIPP sample is smaller than the CPS and NHIS samples, does not contain a representative sample from each state, and thus cannot be used to make state-level estimates.²
• NSAF: Recently started by the Urban Institute as a part of their Assessing the New Federalism project, the NSAF collects information on the economic, health, and social characteristics of children and nonelderly adults in 13 states. It also provides nationwide estimates by surveying an additional nationally representative sample. The NSAF oversamples the populations below 200% of the Federal Poverty Level and provides data on health care access, utilization, health status, and various health insurance estimates (full year, point-in-time, and ever insured during the year). However, it is a new survey started in 1997 and conducted approximately every other year; therefore it cannot measure annual trends nor historical trends prior to 1997.³
• CTS: Administered by the Center for Studying Health System Change, CTS tracks changes in health systems relating to providers and consumers. One of its components, the household survey, provides information on health insurance status, health care access, utilization, delivery, quality and cost and focuses on 60 selected communities while providing national estimates. Regarding health insurance status, CTS can provide point-in-time and full year uninsured estimates, although the Center has not published the latter. While the 1996 survey could not determine if individuals had two different types of coverage simultaneously, subsequent surveys will be able to do so. CTS is also a new survey conducted every other year and cannot provide annual or historical time series. Like most household surveys, the CTS sample does not contain a representative sample from each state and cannot be used to make state-level estimates.

The Estimates

The following table presents the uninsured estimates from the major surveys discussed above. As the table shows, there is significant variability in the number of uninsured from survey to survey.

Table 1. Uninsured Estimates from Various National Surveys

*These estimates were obtained directly from NCHS, which usually only publishes a point-in-time uninsured estimate for the nonelderly population. The above NHIS point-in-time estimate is slightly higher than that published in Health, United States, 1999 because it refers to the entire population, rather than just the nonelderly population.

Some Reasons for Differences Among Estimates

As is clear from the above table, the number of uninsured can vary widely from survey to survey. There are many variables that contribute to variance in survey estimates, such as general survey design and data collection methods. Below is a discussion of the major reasons that most likely explain the variance in uninsured numbers.⁴ (See Table 2 at the end of this paper for a summary of key survey differences)

1. Survey Design Differences

The Length of Time Uninsurance is Measured. The most obvious variable in the above table is the length of time used to measure an insured spell. This time dimension is particulaly important because health insurance status is relatively volatile and estimates will vary with the length of time coverage is examined. As more time passes, more people will experience a lapse in health coverage. Thus, the total number of people who experience a period of uninsurance over the course of an entire year will be greater than the number of uninsured at a given point in time (such as the time of the survey interview). Furthermore, of the people who experience a lapse in coverage, most will have short spells without coverage while fewer will have spells that last a year or longer. Therefore, the full-year uninsured estimate (the number of people who were uninsured for a full twelve moths) will generally be smaller than the point-in-time and ever-uninsured-during-the-year estimates. Therefore, when analyzing survey results, it is important to understand what period of time the data are measuring.

The CPS Controversy. Because the CPS is the most widely cited source of insurance coverage estimates, it is important to note a fact that can be readily seen in the above table: the CPS estimate of the full-year uninsured, 44.3 million, is far higher than the full-year estimates provided by other surveys. In fact, the CPS full-year estimate looks more in-line with other surveys’ point-in-time estimates. Given this higher estimate, many analysts hypothesize that a number of respondents to the CPS actually provide point-in-time information ( i.e., information about their status at the time they are participating in the survey (March) or about their status at the end of the previous calendar year) despite the fact that the survey questions ask about the entire previous calendar year. How many respondents misinterpret the relevant questions in this manner, if any, is unknown, but this uncertainty may make it more difficult to interpret and compare data on the uninsured.

Recall Periods. A second important difference between surveys is the length of the recall period. Most survey methodologists believe that the longer the recall period, the less accurate the answers will be to questions about status in the past. For the CPS, the respondent is required to remember the previous 14 months since the health supplement is conducted in March following the calendar year to which the questions pertain. All other surveys except the CPS ask about coverage at the time of the interview. When determining health insurance status over the year, the recall period is up to 3-5 months for MEPS, up to four months for SIPP, and up to 12 months for NHIS and NSAF.

How “Insurance” Is Defined. The way insurance is defined in each survey determines who is categorized as “insured.” MEPS defines private insurance as coverage for hospital and physician services, thereby eliminating single service, serious and dread disease, workers’ compensation, accident, and disability policies from counting as “coverage.” NHIS does not include single service plans as private insurance except for single service hospital coverage. CPS, SIPP, CTS and NSAF instruct interviewers to not count single service plans (such as dental plans) as private insurance, but some single service coverage may get misreported as comprehensive coverage. CTS is also the only one of these surveys that classifies care from the Indian Health Service as coverage.

How Respondents Are Asked about Insurance. To determine if a person is uninsured, surveys use one of two methods: they ask respondents directly if they lack insurance coverage or they classify individuals as uninsured if they do not affirmatively indicate that they have coverage. MEPS, NHIS, NSAF and CTS directly ask respondents whether they lack insurance coverage, while SIPP and CPS do not. While the difference in the two approaches may seem subtle, some researchers believe that surveys which fail to verify a lack of coverage will overcount the uninsured. The theory is that this verification question prompts some people who did not indicate any insurance coverage to rethink their status and indicate coverage that they previously forgot to mention. Because of this issue, NHIS added a verification question in 1997 and CPS will add this component in 2000. It should be noted that the effect of this question has not been uniform in all the surveys which use it: in CTS and NSAF, asking the verification question has resulted in lower uninsured estimates while the effect on the 1997 NHIS estimates appeared to be negligible.

Focus of Survey. Each survey has a different focus and asks questions about health and health insurance with differing degrees of detail. The CPS is primarily a labor force survey, with some health insurance questions added onto the end. In contrast, the other surveys have a clearer focus on health and health insurance. While difficult to verify, some believe that a focused health survey is more likely to elicit accurate responses regarding health insurance because survey questions on health status and health care utilization will lead to better recall regarding the insurance coverage that was needed to pay for treatment. For example, when comparing CPS and SIPP, Census Bureau officials have hypothesized that one of the reasons SIPP may be better suited to measure health insurance status is that the health questions are more detailed and that they actually ask to see insurance cards.⁵  Similarly, the MEPS survey asks a number of questions about health status and medical care utilization before asking about insurance status.

2. Differences in Timeliness of Data.

Because of different lag times between data gathering and data availability, it is often the case that results cited at the same time actually reflect data gathered in different years. For example, the most recent data available from the CPS were collected in March of 1999 and reflect 1998 coverage rates. The most recent data available from the NHIS were collected in 1997 and reflect 1997 coverage rates. The most recent MEPS data on insurance coverage were collected in 1996 and reflect 1996 coverage rates. NSAF data collected in 1997 were available in 1999, and CTS data collected in 1996 were available in 1998. Therefore, when comparing estimates of the uninsured, it is important to note which year the survey covers.

3. Data Reporting.

Surveys often make available public data files which contain the raw survey data, allowing organizations to publish their own uninsured estimates. Various researches and organizations have adopted different methods to adjust the raw survey data to meet their specific needs. For example, some will adjust the CPS to account for Medicaid under-reporting while others will adjust income measures to better simulate eligibility for government programs. These researcher-specific adjustments can result in different estimates derived from the same survey, particularly for population subgroups, such as the number of uninsured below poverty or the number of uninsured children eligible for Medicaid. Therefore, the uninsured estimates from CPS published by the U.S. Census Bureau have at times differed from those published by other organizations because of differences in the way the data are compiled.⁶

Table 2. Selected Differences Between Surveys’ Uninsured Estimates

Conclusion

It is clear that the estimates of the uninsured may vary depending upon the data source and data adjustments. The decision of which survey to use may depend on the purpose of the analysis. For credible state-level estimates, the CPS is the only source for all 50 states while NSAF produces them for the 13 states it studies in depth. Larger sample sizes enable CPS, NHIS and NSAF to produce more reliable estimates for subgroups of the population (i.e. children, low-income workers, etc.). MEPS and SIPP are the best sources for examining changes in individuals’ insurance status over time and NHIS, MEPS, NSAF, CTS and SIPP can provide point-in-time estimates of the uninsured.

Despite the differences that distinguish these surveys, the estimates derived from each paint a relatively consistent picture of health coverage rates in the United States. Critical policy concerns such as the disparity in coverage rates across income groups and the number of individuals that lack coverage are clearly apparent in the empirical data from all six surveys.

Notes

1. CPS switched to a computer-assisted personal interview (CAPI) system in 1994. Also, in 1995, changes were made to the health section of the CPS March supplement such as reordering the questions and changing some questions about private insurance. In 1997, major changes were made to the NHIS, such as no longer imputing Medicaid to AFDC recipients, switching to a CAPI system and redesigning the survey to include more specific health insurance questions. Survey instrument adjustments need to be taken into account when looking at a historical time series because new survey designs can produce changes in estimates that may not reflect actual changes in the insurance coverage of the population.

2. In 1996, SIPP started asking about coverage during the month of the interview, in addition to asking about coverage during the months prior to the interview. In 1994, it only asked about coverage during the months prior to the interview.

3. NSAF has been fielded in 1997 and 1999 but the next survey will be in 2002.

4. Also affecting estimates are factors such as the target population (e.g. U.S. civilian noninstitutionalized), the survey mode (face-to-face interview vs. telephone), survey response rates, and nonresponse rates to health insurance coverage questions, which are not discussed in this paper but should be noted.

5. Lewis, Kimball, Marilyn Ellwood and John Czajka. “Counting the Uninsured: A Review of the Literature.” Assessing the New Federalism: Occasional Paper Number 8. July 1998. Pg 15. http://newfederalism.urban.org/html/occ8.htm

6. Previous version of this memo contained a section called “Medicaid Adjustments,” which explained how some surveys, most notably the CPS, assign Medicaid coverage to individuals who also receive welfare assistance even if the survey respondents do not report Medicaid coverage. Historically, this assignment of Medicaid, called an “imputation,” was done because a significant percentage of respondents who indicated they had AFDC (the pre-welfare reform cash assistance program) failed to report Medicaid, even though people who received AFDC automatically received Medicaid by law. The Census Bureau continues to assign Medicaid to those who indicate they receive cash assistance through the new TANF program, though Medicaid and welfare eligibility have been delinked. Census continues to make this imputation because it appears States still provide Medicaid to the vast majority of TANF cash recipients. It was decided to drop this section from the memo because some surveys which do not assign Medicaid to welfare recipients find a higher number of Medicaid enrollees than the CPS. Therefore, it is no longer clear that the Medicaid imputation is a major factor in explaining why survey estimates vary. Nonetheless, Medicaid imputation is an important survey design difference that analysts should be aware of.

7. The CTS can also determine the health insurance status of respondents immediately proceeding their insurance status at the time of the interview. But unlike the SIPP and MEPS, it can not determine each type of insurance coverage held during the course of the entire year, should there be more than two.

Additional Information

Websites

Agency for Healthcare Quality and Research. Medical Expenditure Panel Survey.http://www.meps.ahrq.gov

Center for Studying Health System Change. Community Tracking Study. http://www.hschange.com/tracking.html

National Center for Health Statistics. National Health Interview Survey.http://www.cdc.gov/nchs/nhis.htm

Urban Institute. National Survey of American Families. http://newfederalism.urban.org/nsaf/index.htm

U.S. Census Bureau. Health Insurance Data, Current Population Survey (CPS) and Survey of Income and Program Participation (SIPP).http://www.census.gov/hhes/www/hlthins.html

U.S. Census Bureau. Survey of Income and Program Participation Main Page. http://www.sipp.census.gov/sipp

U.S. Census Bureau. Current Population Survey Main Page. http://www.bls.census.gov/cps/cpsmain.htm

Articles

Bennefield, Robert. “A Comparative Analysis of Health Insurance Coverage Estimates: Data From CPS and SIPP.” Presentation at the 1996 Joint Statistical Meetings of the American Statistical Association, 1996c.

Bennefield, Robert. “Who Loses Coverage and for How Long?” Current Population Reports, P70-54. Washington, DC: Census Bureau, May 1996b.

Center for Studying Health System Change and Mathematica Policy Research, Inc. “Estimates of Health Insurance Coverage in the Community Tracking Study and the Current Population Survey.” Technical Publication 16, November 1998. http://www.hschange.com/tech16/8525_toc.html

Kenney, Genevieve et al. “The National Survey of America’s Families: An Overview of the Health Policy Component.” Inquiry. Fall 1999. Vol. 36, pg. 353-362.

Lewis, Kimball, Marilyn Ellwood and John Czajka. “Counting the Uninsured: A Review of the Literature.” Assessing the New Federalism: Occasional Paper Number 8. July 1998. Pg 15. http://newfederalism.urban.org/html/occ8.htm

Monheit, Alan. “Underinsured Americans: A Review.” Ann. Rev. Public Health. 1994. Vol. 15, pg. 461-85.

Rajan, Shruti, Stephen Zuckerman, and Niall Brennan. “Verifying Insurance Coverage: Impact on Measuring the Uninsured with NSAF.” Assessing the New Federalism, The Urban Institute. August 1999.