Income Data for Policy Analysis: A Comparative Assessment of Eight Surveys. Program participation

12/23/2008

How fully and accurately a survey identifies the participants in an entitlement or other means-tested program determines how useful that survey may be for policy analysis of that program and related programs. In comparing the surveys with respect to their estimates of program participation, we focus on welfare (cash assistance) and Food Stamps, SSI, and Medicaid.32 As a rule, surveys underestimate the numbers of participants in means-tested programs, so in comparing estimates of participants across surveys, “more” is generally better.

Differences in estimates of participants in welfare or Food Stamps, SSI, and Medicaid are quite substantial across the five surveys. For each program, SIPP finds the most participants by a wide margin over any other survey (Table IV.22). For example, SIPP finds 31.4 million persons (or 11.2 percent of the population) in families receiving welfare or Food Stamps at any time during 2002. The ACS is second with 24.3 million or 8.8 percent, followed by the CPS and MEPS with 20.5 million (7.3 percent) and 20.3 million (7.1 percent), respectively. The NHIS identifies fewer than half as many as SIPP—just 14.3 million or 5.0 percent of the population.33

TABLE IV.22

ESTIMATES OF PROGRAM PARTICIPANTS: FIVE SURVEYS
Estimate CPS ACS SIPP MEPS NHIS
Millions of Persons
All Persons 282.55 277.69 281.08 283.30 283.71
Program
Welfare or Food Stamps 20.50 24.33 31.41 20.23 14.29
SSI 4.88 4.55 8.38 6.40 5.50
Medicaid
Ever in prior calendar year 32.86 NA  48.11 41.23 NA 
Current month NA  NA  33.28 34.96 29.90

 

ESTIMATES OF PROGRAM PARTICIPANTS: FIVE SURVEYS (continued)
Estimate CPS ACS SIPP MEPS NHIS
Percent of the Population
All Persons 100.00 100.00 100.00 100.00 100.00
Program
Welfare or Food Stamps 7.3 8.8 11.2 7.1 5.0
SSI 1.7 1.6 3.0 2.3 1.9
Medicaid
Ever in prior calendar year 11.6 NA  17.1 14.6 NA
Current month NA  NA  11.8 12.3 10.5

Source:   Mathematica Policy Research, from tabulations of the 2003 CPS ASEC supplement, the 2002 ACS, the 2001 SIPP panel, the 2002 Full-year Consolidated MEPS-HC, and the 2003 NHIS. Note:  Except where noted, participation is ever during 2002 or the previous 12 months (ACS).

For SSI, the CPS and ACS find the fewest participants among the five surveys. SIPP finds 8.38 million or 3.0 percent of the population, followed by MEPS with 6.4 million (or 2.3 percent) and NHIS with 5.5 million (or 1.9 percent). The CPS and ACS find 4.9 million (1.7 percent) and 4.6 million (1.6 percent), respectively.

Our comparisons of Medicaid enrollment utilize two different reference periods in order to maximize the possible comparisons across surveys and to make a point about reporting error. The CPS asks respondents if they were ever enrolled in Medicaid during the previous calendar year (2002) while the NHIS asks respondents if they are enrolled at the time of the survey (January through December 2003). Both SIPP and MEPS capture Medicaid enrollment on a monthly basis, so we can compare estimates of persons ever enrolled in 2002 with the CPS and compare enrollment in December 2002 to the NHIS. The ACS did not include a question on Medicaid enrollment until January 2008.

SIPP finds 48.1 million or 17.1 percent of the population ever enrolled in Medicaid during the 2002 calendar year while MEPS finds 41.2 million or 14.6 percent. The CPS is well behind with 32.9 million or 11.6 percent of the population. MEPS finds more Medicaid enrollees in December 2002 than SIPP, with 35.0 million or 12.3 percent of the population compared to SIPP’s 33.3 million or 11.8 percent. NHIS finds an average monthly enrollment of 29.9 million or 10.5 percent of the population in 2003. It is noteworthy that the MEPS and SIPP estimates of Medicaid enrollment in December 2002 exceed the CPS estimate of persons who were ever enrolled in 2002. This illustrates a well-known problem with CPS estimates of Medicaid—namely, that the survey’s estimates of people who were ever enrolled during a year bear a closer resemblance to panel surveys’ estimates of persons enrolled at a single point in time than to estimates of persons ever enrolled in a year. A popular interpretation is that CPS respondents are answering the question about their Medicaid enrollment in the prior year with their current enrollment.

For the overlapping populations participating in either welfare or Food Stamps (or both), we compared the five surveys’ estimates of participants by quintile of family income. On doing so, we find that the SIPP’s margin over the CPS grows as the quintile increases (Table IV.23). While SIPP finds a third more participants than the CPS among persons in the bottom quintile and 46 percent more in the second quintile, SIPP finds twice as many in the third quintile, more than four times as many in the fourth quintile, and nearly six times as many in the top quintile. The ACS shows a similar pattern relative to the CPS, and both SIPP and ACS show progressively more beneficiaries than MEPS or NHIS as the income quintile rises as well. This may be indicative of a problem with the reporting or, more likely, the imputation of welfare and Food Stamp Program benefits in the SIPP and ACS. At the same time, however, if both surveys are more effective at identifying welfare and Food Stamp Program beneficiaries than other surveys, part of their success may lie in eliciting reports of participation from people who might be least inclined to report their participation—such as those who received benefits for a brief period when their incomes were much lower than they were for the rest of the year.

Reports of participation in welfare or Food Stamps or SSI (the latter among family heads only) in the PSID compare to those that were obtained in the CPS (Table IV.24). Focusing on participants expressed as percentages of the population, we find that participants in the PSID, the full CPS, and CPS-X were 7.3 percent of the population compared to SIPP’s 11.2 percent. Likewise, family heads who received SSI were 0.9 percent of the population in the PSID, the full CPS, and CPS-X and 1.0 percent in the ACS. The corresponding participation rate in the SIPP was 1.6 percent.

TABLE IV.23

PERSONS IN FAMILIES WITH WELFARE AND/OR FOOD STAMPS BY QUINTILE OF FAMILY INCOME: FIVE SURVEYS
Income Estimate CPS ACS SIPP MEPS NHIS
Thousands of Persons
All Participants 20,496 24,325 31,406 20,226 21,990
Family Income Quintile
Lowest 13,562 14,879 18,001 13,949 14,783
Second 4,461 4,854 6,498 4,512 4,355
Third 1,748 2,396 3,520 1,189 1,685
Fourth 493 1,273 2,044 393 719
Highest 233 923 1,343 183 447
Sum through Four Quintiles 20,263 23,402 30,063 20,043 21,543

 

PERSONS IN FAMILIES WITH WELFARE AND/OR FOOD STAMPS BY QUINTILE OF FAMILY INCOME: FIVE SURVEYS (continued)
Income Estimate CPS ACS SIPP MEPS NHIS
Percent of CPS
All Participants 100.00 118.7 153.2 98.7 107.3
Family Income Quintile
Lowest 100.0 109.7 132.7 102.9 109.0
Second 100.0 108.8 145.7 101.1 97.6
Third 100.0 137.0 201.3 68.0 96.4
Fourth 100.0 258.4 414.8 79.7 145.9
Highest 100.0 396.8 577.2 78.6 192.2
Sum through Four Quintiles 100.0 115.5 148.4 98.9 106.3

Source: Mathematica Policy Research, from tabulations of the 2003 CPS ASEC supplement, the 2002 ACS, the 2001 SIPP panel, the 2002 Full-year Consolidated MEPS-HC, and the 2003 NHIS. Note: Participation is ever during 2002 or the previous 12 months (ACS).

TABLE IV.24

ESTIMATES OF PROGRAM PARTICIPANTS: PSID AND CENSUS BUREAU SURVEYS
Estimate CPS ACS SIPP PSID CPS-X1
Millions of Persons
All Persons 282.55 277.69 281.08 261.45 274.44
(Program)Welfare or Food Stamps2 20.50 24.32 31.41 19.19 19.91
(Program)SSI among family heads 2.44 2.64 4.45 2.45 2.44
 ( (Program)Medicaid) Ever in prior calendar year 32.86 NA 48.11 16.003 32.01
 ( (Program)Medicaid)  Current month NA  NA 33.28 NA  NA 

 

ESTIMATES OF PROGRAM PARTICIPANTS: PSID AND CENSUS BUREAU SURVEYS(continued)
Estimate CPS ACS SIPP PSID CPS-X1
Percent of the Population
All Persons 100.0 100.0 100.0 100.0 100.0
(Program)Welfare or Food Stamps2 7.3 8.8 11.2 7.3 7.3
(Program)SSI among family heads 0.9 1.0 1.6 0.9 0.9
 ((Program)Medicaid) Ever in prior calendar year 11.6 NA 17.1 6.1 11.7
((Program)Medicaid)  Current month NA NA 11.8 NA NA

Source:  Mathematica Policy Research, from tabulations of the 2003 CPS ASEC supplement, the 2002 ACS, the 2001 SIPP panel, and the 2003 PSID. Note: Except where noted, participation is ever during 2002 or the previous 12 months (ACS).

1.The CPS-X estimates exclude all unrelated subfamilies and most secondary individuals (except unmarried partners of the householder) to mimic the population controls applied to the PSID.

2. Persons are counted if the head of the family is receiving welfare or if anyone in the family is receiving food stamps.

3. Persons with Medicaid and no other coverage in 2001 and 2002 and at least one month of health insurance coverage in 2002.

The PSID asks its respondents about their Medicaid participation over the prior two calendar years and does not obtain separate reports by year. We approximated a measure of ever enrollment in 2002 by identifying persons who were ever enrolled in Medicaid in 2001 or 2002, had no other coverage during that period, and were uninsured for no more than 11 months of 2002. This yielded a very low enrollment estimate—only 16.0 million persons or 6.1 percent of the population or just a third of the SIPP estimate of 48.1 million or 17.1 percent and barely half of the CPS estimate of 11.6 percent. Had we included PSID respondents who reported coverage in addition to Medicaid during the two-year period (and who might have been covered by something other than Medicaid during 2002), we would have increased the PSID estimate by only a small amount.

When reported welfare or Food Stamp Program beneficiaries are distributed by quintile of family income, we find that the PSID estimates fall off more rapidly than the CPS estimates as the quintile increases (Table IV.25). In the lowest quintile the beneficiaries identified in the PSID are 97.8 percent of the number identified in the full CPS (and a larger fraction of those identified in CPS-X). This drops to 93.7 percent in the second quintile, with no change relative to CPS-X, and then 66.1 percent in the third quintile, where the PSID falls relative to CPS-X as well. In the fourth quintile the PSID estimate is comparable to both CPS estimates, but in the highest quintile the PSID estimate is only 51 percent of the full CPS estimate and, not shown, only 56 percent of the CPS-X estimate. We find it interesting and perhaps informative that the PSID should show the same pattern of declining enrollment by quintile when compared to the CPS that we saw when comparing the CPS to SIPP. Once again, this could reflect growing reluctance to report prior welfare or Food Stamp receipt as income increases, but it could also reflect differences in imputation across the surveys. The PSID does not make use of the hot deck imputation methods employed in the CPS and SIPP and, therefore, the PSID results may reflect actual reporting patterns more closely than they do in the other surveys.

TABLE IV.25

PERSONS IN FAMILIES WITH WELFARE AND/OR FOOD STAMPS BY QUINTILE OF FAMILY INCOME: PSID AND CENSUS BUREAU SURVEYS
Income Estimate CPS ACS SIPP PSID CPS-X1
Thousands of Persons
All Participants 20,496 24,325 31,406 19,186 19,906
Family Income Quintile(Lowest) 13,562 14,879 18,001 13,264 13,389
Family Income Quintile(Second) 4,461 4,854 6,498 4,182 4,217
Family Income Quintile(Third) 1,748 2,396 3,520 1,155 1,606
Family Income Quintile(Fourth) 493 1,273 2,044 466 481
Family Income Quintile(Highest) 233 923 1,343 119 212
Sum through Four Quintiles 20,263 23,402 30,063 19,066 19,693

 

PERSONS IN FAMILIES WITH WELFARE AND/OR FOOD STAMPS BY QUINTILE OF FAMILY INCOME: PSID AND CENSUS BUREAU SURVEYS(continued)
Income Estimate CPS ACS SIPP PSID CPS-X1
Percent of CPS
All Participants 100.0 118.7 153.2 93.6 97.1
Family Income Quintile(Lowest) 100.0 109.7 132.7 97.8 98.7
Family Income Quintile(Second) 100.0 108.8 145.7 93.7 94.5
Family Income Quintile(Third) 100.0 137.0 201.3 66.1 91.9
Family Income Quintile(Fourth) 100.0 258.4 414.8 94.6 97.6
Family Income Quintile(Highest) 100.0 396.8 577.2 51.2 91.3
Sum through Four Quintiles 100.0 115.5 148.4 94.1 97.2

Source:   Mathematica Policy Research, from tabulations of the 2003 CPS ASEC supplement, the 2002 ACS, the 2001 SIPP panel, and the 2003 PSID

1. The CPS-X estimates exclude all unrelated subfamilies and most secondary individuals (except unmarried partners of the householder) to mimic the population controls applied to the PSID.

 

View full report

Preview
Download

"report.pdf" (pdf, 4.33Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®