ASPE ISSUE BRIEF

Contents

• Introduction
• General Survey Descriptions
• The Estimates
• Some Reasons for Differences Among Estimates
  1. Survey Design Differences
  2. Differences in Timeliness of Data
  3. Data Reporting
• Conclusion

EndnotesAdditional Information

Introduction

The number of uninsured individuals in the United States has been an important policy concern for many years. The four federal surveys that are major sources of data on the uninsured have played an important role in informing policy on this issue. These surveys: the Current Population Survey (CPS), the Medical Expenditure Panel Survey (MEPS), the National Health Interview Survey (NHIS), and the Survey of Income and Program Participation (SIPP) can each provide useful estimates of the number of uninsured individuals during a particular period of time, and in some cases, at a point in time. Both MEPS and SIPP also provide information on changes in the insurance status of individuals over time.

Why are there so many surveys which measure health insurance? Actually, each of the surveys measures far more than insurance status. Surveys which measure insurance status actually address multiple topics and each has different strengths. The CPS, primarily a survey related to employment, has the best source of income data and is large enough to yield state data. The NHIS, the broadest survey of health, is useful in analysis of disease prevalence and health risks in the population. The MEPS provides the greatest depth regarding expenditures and utilization. The SIPP can measure monthly changes in income and employment status. Because health insurance is so important, it is collected on each of these surveys so that it can be related to other content that is unique to those surveys. What follows is a more complete description of each survey.

[ Go to Contents ]

General Survey Descriptions

CPS:  Conducted by the U.S. Census Bureau, the CPS is a monthly survey designed to collect data on the populations employment status. The Annual Social and Economic Supplement to the CPS asks additional questions about health insurance coverage. The CPS data are widely used because the CPS has the largest household survey sample size, is designed to produce credible state-level estimates (less populous states may require 3 years of CPS data to produce such estimates), is available on a timely basis, and provides information on coverage rates for socio-demographic subgroups of the population. However, as with most surveys, the CPS has been subject to questions regarding over- and under-reporting. Specifically, it is thought that the CPS over-counts the number of individuals who have been uninsured for an entire year, possibly because respondents answer based on current, rather than previous, coverage status. In addition, Medicaid coverage status is likely under-reported.⁽¹⁾

MEPS:  Administered by the Agency for Healthcare Research and Quality (AHRQ), MEPS is a two-year panel survey that collects information about health care use and expenditures, and the dynamics of health insurance and delivery systems. MEPS data on an individuals health insurance status are collected several times a year and have the capacity to produce a variety of estimates of health insurance coverage, including point-in-time, monthly, and annual estimates. As a longitudinal survey, MEPS is able to examine health insurance dynamics, including changes in coverage and spells without coverage. MEPS data can be used to analyze the relationship between insurance status and a variety of individual and household characteristics, including use of and expenditures for health care services. Data on sources of payment for health care are collected and can be used to add greater accuracy and precision to the insurance status estimates. The MEPS sample is smaller than the CPS and NHIS samples and does not contain a representative sample from each state, and thus cannot be used to make state-level estimates.

NHIS:  Administered by the National Center for Health Statistics (NCHS), the NHIS is a continuous household interview survey of the civilian noninstitutionalized population. The NHIS collects detailed information on insurance status, type of coverage, measures of health status, health care utilization, and socio-demographic characteristics of members of sampled households. It provides estimates of the number of persons uninsured at a point-in-time, uninsured for a full-year, and ever uninsured during the year. The NHIS also collects data on illnesses, injuries, activity limitations, chronic conditions, health behaviors, and other health topics, which can be linked to characteristics such as insurance status. Starting in 2001, NCHS began publishing quarterly early reports that provide insurance data only six months after a given quarter, which is the most up-to-date data of any survey. Like MEPS and SIPP, the NHIS cannot provide reliable state estimates, except in the largest states.

SIPP:  Also administered by the Census Bureau, SIPP is a 36- to 48-month longitudinal panel survey that collects information on health insurance status, income, labor force activity and federal program participation. As a longitudinal survey, SIPP data provide the capacity to examine the dynamics of health insurance. It measures the duration of spells without health insurance and provides health insurance estimates for various time periods, such as point-in-time, monthly, annual, or over the full panel. As part of the core data collected in the SIPP, health insurance data can also be linked to other sections of the survey, such as utilization of health care services, child well-being, and disability. The SIPP sample is smaller than the CPS and NHIS samples and does not contain a representative sample from each state, and thus cannot be used to make state-level estimates.⁽²⁾

[ Go to Contents ]

The Estimates

The following table presents the uninsured estimates from the major surveys discussed above. As the table shows, there is significant variability in the number of uninsured from survey to survey.

[ Go to Contents ]

Some Reasons for Differences Among Estimates

As is clear from the above table, the number of uninsured can vary widely from survey to survey. There are many variables that contribute to variance in survey estimates, such as general survey design and data collection methods. Below is a discussion of the major reasons that most likely explain the variance in uninsured numbers.⁽³⁾  (See Table 2 at the end of this paper for a summary of key survey differences)

1. Survey Design Differences

• The Length of Time Uninsurance is Measured. The most obvious variable in the above table is the length of time used to measure an insured spell. This time dimension is particularly important because health insurance status is relatively volatile and estimates will vary with the length of time coverage is examined. As more time passes, more people will experience a lapse in health coverage. Thus, the total number of people who experience a period of uninsurance over the course of an entire year (referred to as ever-uninsured) will be greater than the number of uninsured at a given point in time (such as the time of the survey interview). Furthermore, of the people who experience a lapse in coverage, most will have short spells without coverage while fewer will have spells that last a year or longer. Therefore, the full-year uninsured estimate (the number of people who were uninsured for a full twelve months) will generally be smaller than the point-in-time and ever-uninsured-during-the-year estimates. Therefore, when analyzing survey results, it is important to understand what period of time the data are measuring.
• The CPS Controversy. Because the CPS is the most widely cited source of insurance coverage estimates, it is important to note a fact that can be readily seen in Table 1: the CPS estimate of the full-year uninsured in 2004, 45.8 million, is far higher than the full-year estimates provided by other surveys. In fact, the CPS full-year estimate looks more in-line with other surveys point-in-time estimates. Given this higher estimate, many analysts hypothesize that a number of respondents to the CPS actually provide point-in-time information (i.e., information about their status at the time they are participating in the survey (March) or about their status at the end of the previous calendar year) despite the fact that the survey questions ask about the entire previous calendar year. How many respondents misinterpret the relevant questions in this manner, if any, is unknown, but this uncertainty may make it more difficult to interpret and compare data on the uninsured.
• Recall Periods. A second important difference between surveys is the length of the recall period. Most survey methodologists believe that the longer the recall period, the less accurate the answers will be to questions about status in the past. For the CPS, the respondent is required to remember the previous 13 to 15 months since the health supplement is conducted in March following the calendar year to which the questions pertain. All other surveys except the CPS ask about coverage at the time of the interview. When determining health insurance status over the year, the recall period is up to 3-5 months for MEPS, up to four months for SIPP, and up to 12 months for NHIS.
• How Insurance Is Defined. The way insurance is defined in each survey determines who is categorized as insured. MEPS defines private insurance as coverage for hospital and physician services, thereby eliminating serious and dread disease, workers compensation, accident, and disability policies from counting as coverage. As well, MEPS does not include single service plans (such as dental plans) as private insurance except for single service hospital coverage. CPS and SIPP instruct interviewers to not count single service plans as private insurance, but some single service coverage may get misreported as comprehensive coverage.
• How Respondents Are Asked about Insurance. To determine if a person is uninsured, surveys use one of two methods: they ask respondents directly if they lack insurance coverage or they classify individuals as uninsured if they do not affirmatively indicate that they have coverage. CPS and NHIS directly ask respondents whether they lack insurance coverage, while MEPS and SIPP do not. While the difference in the two approaches may seem subtle, some researchers believe that surveys which fail to verify a lack of coverage will over count the uninsured. The theory is that this verification question prompts some people who did not indicate any insurance coverage to rethink their status and indicate coverage that they previously forgot to mention. Because of this issue, NHIS added a verification question in 1997 and CPS added this component in 2000. It should be noted that the effect of this question has not been uniform in all the surveys which use it: in several privately funded surveys that are also sources of data on the uninsured, the National Survey of Americas Families (NSAF) and the Community Tracking Study (CTS), asking the verification question has resulted in lower uninsured estimate, while the effect on the 1997 NHIS estimates appeared to be negligible.
• Focus of Survey. Each survey has a different focus and asks questions about health and health insurance with differing degrees of detail. The CPS is primarily a labor force survey, with some health insurance questions added onto the end. In contrast, the other surveys have a clearer focus on health and health insurance. While difficult to verify, some believe that a focused health survey is more likely to elicit accurate responses regarding health insurance because survey questions on health status and health care utilization will lead to better recall regarding the insurance coverage that was needed to pay for treatment. For example, when comparing CPS and SIPP, Census Bureau officials have hypothesized that one of the reasons SIPP may be better suited to measure health insurance status is that the health questions are more detailed and that they actually ask to see insurance cards.⁽⁴⁾ Similarly, the MEPS survey asks a number of questions about health status and medical care utilization before asking about insurance status.

2. Differences in Timeliness of Data

Because of different lag times between data gathering and data availability, it is often the case that results cited at the same time actually reflect data gathered in different years. For example, the most recent data available from the CPS were collected in March of 2005 and reflect 2004 coverage rates, the most recent data available from the NHIS were collected in 2004 and reflect 2004 coverage rates, and the most recent MEPS data on full year insurance coverage were collected throughout 2003 and into 2004, reflecting 2003 coverage rates. MEPS data for point in time insurance coverage is collected in early 2004 and released in June of 2005. Therefore, when comparing estimates of the uninsured, it is important to note which year the survey covers.

3. Data Reporting

Surveys often make available public data files which contain the raw survey data, allowing organizations to publish their own uninsured estimates. Various researchers and organizations have adopted different methods to adjust the raw survey data to meet their specific needs. For example, some will adjust the CPS to account for Medicaid under-reporting while others will adjust income measures to better simulate eligibility for government programs. These researcher-specific adjustments can result in different estimates derived from the same survey, particularly for population subgroups, such as the number of uninsured below poverty or the number of uninsured children eligible for Medicaid. Therefore, the uninsured estimates from CPS published by the U.S. Census Bureau have at times differed from those published by other organizations because of differences in the way the data are compiled.⁽⁵⁾

[ Go to Contents ]

Conclusion

It is clear that the estimates of the uninsured may vary depending upon the data source and data adjustments. The decision of which survey to use may depend on the purpose of the analysis. For credible state-level estimates, the CPS is the only source for all 50 states. Larger sample sizes enable CPS and NHIS to produce more reliable estimates for subgroups of the population (i.e. children, low-income workers, etc.). MEPS and SIPP are the best sources for examining changes in individuals insurance status over time and NHIS, MEPS, and SIPP can provide point-in-time estimates of the uninsured.

Despite the differences that distinguish these surveys, the estimates derived from the surveys in total paint a relatively consistent picture of health coverage rates in the United States. While the overall number of uninsured varies from survey to survey, rising and falling trends in the estimates are consistent between the surveys. In addition, uninsured estimates among population subgroups also tend to be consistent. For example, all surveys show that the uninsured are concentrated among low-income individuals, and all surveys show that Hispanics have the highest uninsured rate among the racial and ethnic groups. Therefore, the surveys can still help policy makers tackle critical policy concerns related to health insurance coverage.

[ Go to Contents ]

Endnotes

1. Previous version of this memo contained a section called Medicaid Adjustments, which explained how some surveys, most notably the CPS, assign Medicaid coverage to individuals who also receive welfare assistance even if the survey respondents do not report Medicaid coverage. Historically, this assignment of Medicaid, called an imputation, was done because a significant percentage of respondents who indicated they had AFDC (the pre-welfare reform cash assistance program) failed to report Medicaid, even though people who received AFDC automatically received Medicaid by law. The Census Bureau continues to assign Medicaid to those who indicate they receive cash assistance through the new TANF program, though Medicaid and welfare eligibility have been delinked. Census continues to make this imputation because it appears States still provide Medicaid to the vast majority of TANF cash recipients. It was decided to drop this section from the memo because some surveys which do not assign Medicaid to welfare recipients find a higher number of Medicaid enrollees than the CPS. Therefore, it is no longer clear that the Medicaid imputation is a major factor in explaining why survey estimates vary. Nonetheless, Medicaid imputation is an important survey design difference that analysts should be aware of.

2. In 1996, SIPP started asking about coverage during the month of the interview, in addition to asking about coverage during the months prior to the interview. In 1994, it only asked about coverage during the months prior to the interview.

3. Also affecting estimates are factors such as the target population (e.g. U.S. civilian noninstitutionalized), the survey mode (face-to-face interview vs. telephone), survey response rates, and nonresponse rates to health insurance coverage questions, which are not discussed in this paper but should be noted.

4. Lewis, Kimball, Marilyn Ellwood and John Czajka. Counting the Uninsured: A Review of the Literature. Assessing the New Federalism: Occasional Paper Number 8. July 1998. Pg 15. http://newfederalism.urban.org/html/occ8.htm

5. CPS switched to a computer-assisted personal interview (CAPI) system in 1994. Also, in 1995, changes were made to the health section of the CPS March supplement such as reordering the questions and changing some questions about private insurance. In 1997, major changes were made to the NHIS, such as no longer imputing Medicaid to AFDC recipients, switching to a CAPI system and redesigning the survey to include more specific health insurance questions. Survey instrument adjustments need to be taken into account when looking at a historical time series because new survey designs can produce changes in estimates that may not reflect actual changes in the insurance coverage of the population.

Additional Information

Websites

Agency for Healthcare Quality and Research, Medical Expenditure Panel Survey:http://www.meps.ahrq.gov

Center for Studying Health System Change, Community Tracking Study:http://www.hschange.com/tracking.html

Economic Research Initiative on the Uninsured, University of Michigan:http://www.umich.edu/eriu

National Center for Health Statistics, National Health Interview Survey:http://www.cdc.gov/nchs/nhis.htm

Urban Institute, National Survey of American Families:http://newfederalism.urban.org/nsaf/index.htm

U.S. Census Bureau. Health Insurance Data, Current Population Survey (CPS) and Survey of Income and Program Participation (SIPP):http://www.census.gov/hhes/www/hlthins.html

U.S. Census Bureau, Survey of Income and Program Participation Main Page:http://www.sipp.census.gov/sipp

U.S. Census Bureau, Current Population Survey Main Page:http://www.bls.census.gov/cps/cpsmain.htm

Articles

Bennefield, Robert. A Comparative Analysis of Health Insurance Coverage Estimates: Data From CPS and SIPP. Presentation at the 1996 Joint Statistical Meetings of the American Statistical Association, 1996c.

Bennefield, Robert. Who Loses Coverage and for How Long? Current Population Reports, P70-54. Washington, DC: Census Bureau, May 1996b.

Bhandari, Shailesh, 2004, "People with Health Insurance: A Comparison of Estimates from two Surveys", Survey of Income and Program Participation Working Paper 243, U.S. Census Bureau. http://www.sipp.census.gov/sipp/workpapr/wp243.pdf

Center for Studying Health System Change and Mathematica Policy Research, Inc. Estimates of Health Insurance Coverage in the Community Tracking Study and the Current Population Survey. Technical Publication 16, November 1998. http://www.hschange.org/CONTENT/172/

Kenney, Genevieve et al. The National Survey of Americas Families: An Overview of the Health Policy Component. Inquiry. Fall 1999. Vol. 36, pg. 353-362.

Lewis, Kimball, Marilyn Ellwood and John Czajka. Counting the Uninsured: A Review of the Literature. Assessing the New Federalism: Occasional Paper Number 8. July 1998. Pg 15. http://newfederalism.urban.org/html/occ8.htm

Monheit, Alan. Underinsured Americans: A Review. Ann. Rev. Public Health. 1994. Vol. 15, pg. 461-85.

Rajan, Shruti, Stephen Zuckerman, and Niall Brennan. Verifying Insurance Coverage: Impact on Measuring the Uninsured with NSAF. Assessing the New Federalism, The Urban Institute. August 1999.

Short, Pamela. Counting and Characterizing the Uninsured, Economic Research Initiative on the Uninsured, ERIU Working Paper 2, Dec. 4, 2001. http://www.umich.edu/~eriu/pdf/wp2.pdf