Using National Survey Data to Analyze Children’s Health Insurance Coverage: An Assessment of Issues. 1. Choice of a Numerator: Survey Versus Administrative Data


Survey estimates of participants in means-tested entitlement programs generally fall well short of the counts reported in program administrative statistics. As a rule, the survey estimates tend to run between 75 and 90 percent of the administrative counts even when the two are rendered as comparable as possible with respect to the universe that they cover. For this reason, it has become a common practice to substitute administrative counts for survey estimates of participants in calculating participation rates for food stamps and AFDC. The choice of a numerator is an issue with respect to Medicaid participation rates as well. Here we discuss a number of considerations that are relevant to using the administrative statistics in this context. The bottom line is that comparability between the administrative and survey data on participation is difficult to establish.

a. Underreporting of Medicaid and Related Program Participation

Table 5 compares CPS estimates of children under 15 who were ever enrolled in Medicaid during 1993, 1994, and 1995 with enrollment statistics reported by the Health Care Financing Administration(now known as Centers for Medicare and Medicaid Services(CMS)) (HCFA(now known as CMS)). While a number of caveats should be addressed in making such a comparison, as we explain in the next subsection, the figures in the table give us a rough sense of how complete the CPS reports of Medicaid enrollment appear to be. In 1994 and 1995 the CPS figures lie between 75 and 76 percent of the HCFA(now known as CMS) estimates versus 83 percent in 1993. The decline in coverage would appear to be due to the CPS’s incomplete capture of a sizable growth in enrollment between 1993 and 1994. Elsewhere, with a more detailed comparison we estimated that the SIPP captured between 85 and 87 percent of Medicaid enrollment among children in FY93 and FY94 (Czajka 1999). The apparent implication, then, is that participation rates will be understated by 13 to 25 percent if we rely exclusively on survey estimates of participation.


Year CPS: Ever Enrolled in Calendar Year HCFA(now known as CMS) Statistics: Ever Enrolled in Fiscal Year CPS Estimate as Percent of HCFA(now known as CMS)
1993 15,165,000 18,348,000 82.7%
1994 14,545,000 19,227,000 75.6%
1995 14,685,000 19,444,000 75.5%
SOURCE: March Current Population Survey, 1994 to 1996, and HCFA(now known as CMS) Medicaid enrollment statistics, FY93 to FY95.

b. Issues in Comparing Survey and Administrative Estimates of Medicaid Enrollment

There are several issues in comparing survey and administrative estimates of Medicaid enrollment to evaluate coverage and, ultimately, to substitute the latter for the former in estimates of participation rates. These include unduplication across states, the existence of state-only programs, the reporting of average monthly versus annual ever enrollment, the limited age detail that is available from published statistics, the inclusion of institutionalized children, concerns about the quality of state Medicaid enrollment data, and retroactive eligibility.

Unduplication Across States. The Medicaid enrollment data published by HCFA(now known as CMS) are based on reports or data files submitted by the states. While researchers have at times expressed concern about duplicate counting of enrollees within states--the classic situation involving someone who is enrolled in Medicaid at the beginning of the fiscal year, leaves the program, then re-enrolls and is assigned a new, unique identification number--within-state duplication has been reduced by administrative improvements. The same cannot be said about duplication across states. People who start the year enrolled in Medicaid in one state, then move to another state and re-enroll, will be counted-- legitimately--in both states’ ever enrollment figures. In the survey data, of course, such people will be counted only once--in the state in which they reside at the time of the interview. There are no data with which to estimate the possible magnitude of this cross-state duplication, which would require matching state administrative files at the person level or matching survey data to these same administrative data. We doubt, however, that such duplication amounts to more than a few percent of the total national caseload reported by HCFA(now known as CMS), although this is purely speculative. About 16 percent of the total U.S. population moves in the course of a year, but only a small fraction of these moves are interstate.

State-Only Programs. A few states (New Jersey, for example) operate what are generally small programs that provide Medicaid coverage to children who do not qualify for federal matching dollars. These children are not included in the enrollment counts reported by HCFA(now known as CMS), but they would presumably report themselves (or be reported) to a survey interviewer as covered by Medicaid. If no allowance is made for their differential inclusion in survey versus federal administrative data, their presence in the survey estimates will contribute to an overestimate of survey coverage of Medicaid enrollees.

Average Monthly versus Annual Ever Enrollment. HCFA(now known as CMS) reports annual (fiscal year) estimates of people ever enrolled in Medicaid by programmatic and demographic characteristics for each state and for the nation as a whole (an aggregate of the state numbers, which may include some duplication). For all people in each state (that is, with no further breakdown), HCFA(now known as CMS) also reports the number enrolled for all 12 months, the number enrolled for less than 12 months, and the total person-months of enrollment among the latter. With these data is possible to calculate the average monthly enrollment--but only for all enrollees. Children and adults cannot be separated. Thus, the most readily available Medicaid administrative data on enrolled children can be used to evaluate only one type of survey estimate of Medicaid coverage: the number of children ever enrolled during a fiscal year. To evaluate survey estimates of Medicaid enrollment at a point in time requires that the researcher make some assumption about how the relationship between ever enrollment in a year and enrollment at a point in time differs between children and adults.

The states can and do produce their own estimates of Medicaid enrollment. They can produce estimates of the number of people enrolled each month by demographic and programmatic characteristics. Such data are not compiled nationally, however. To obtain monthly enrollment estimates, the researcher would have to request these from every state. In practice, then, it is not possible to compare survey and administrative estimates of Medicaid enrollment at a point in time with the same precision that can be done with estimates of enrollment ever in a year.

Age Detail. HCFA(now known as CMS) reports Medicaid enrollees under 21 by the following age groups: infant (under 1), 1 to 5, 6 to 14, and 15 to 20. These age categories do not map exactly to children as commonly defined from survey data: all people under 18 or all people under 19. Because Medicaid enrollment declines over ages 15 to 20, allotting two-thirds of the reported number of enrollees in this age group to the ages 15 to 18 yields too few children. To obtain better administrative estimates of enrolled children, we recommend estimating from survey data the fraction of reported Medicaid enrollees 15 to 20 years of age who are 15 to 18 (or 15 to 17 if that is the needed group) and applying this fraction to the Medicaid administrative data. Another strategy, which we followed in preparing Table 5, is to base the comparison on just those ages that can be matched (that is, 0 through 14) and assume that the same rate of coverage applies to the entire population of child enrollees.(27)

We should note that there is a programmatic definition of “children” used in determining Medicaid eligibility, and that reported counts of “children” in some of the HCFA(now known as CMS) tabulations reflect this definition rather than the purely age-based definition used in survey-based research. Enrollees identified as “children” in HCFA(now known as CMS) reports are a subset of the full age group that would be defined as children in survey-based research. An individual who is under the age of 19 but responsible for a dependent child would be reported as an adult in tabulations of the basis of eligibility.

Institutionalized Children. Administrative estimates of enrollees include some who are institutionalized whereas the surveys that are used to estimate health insurance coverage exclude people in institutions from their sampling frames. Published HCFA(now known as CMS) tabulations do not report institutionalized enrollees by age, so it is not possible to exclude institutionalized children from the administrative counts of Medicaid enrollees--except crudely. This is not a large population, but other things being equal, failing to make some adjustment for its differential treatment in the two estimates will contribute to an underestimate of the survey coverage of Medicaid enrollment.

Quality of State Medicaid Enrollment Data. Researchers have raised concern about the quality of state Medicaid enrollment data. As we noted above, one area of concern was the potential multiple counting of individuals who left the program and re-entered within the same fiscal year, but the widespread use of unique “lifetime” identifiers is eliminating this problem. Indeed, analysis of case record data provides indirect support for this assertion in the form of frequent, identifiable instances of the same individuals exiting and then re-entering Medicaid within the year (Ellwood and Lewis 1999). At the same time, however, the state statistics reported by HCFA(now known as CMS) each year are accompanied by extensive caveats that point out omissions, inconsistencies, and other errors. At a minimum, users of the published enrollment data need to be aware that the data have known imperfections that may require some form of correction before they are used.

Retroactive Eligibility. Under certain circumstances, a Medicaid enrollee’s eligibility may be applied retroactively to cover medical costs that were incurred prior to official enrollment. Survey respondents interviewed just prior to their enrollment may correctly report their status as not covered, but the administrative statistics may later change this status. As a result, the administrative statistics would tend to run slightly higher than the reports obtained from surveys even if the latter were correct at the time they were recorded.