Income Data for Policy Analysis: A Comparative Assessment of Eight Surveys. Aggregate Income by Quintiles


Estimates of aggregate income, for the whole population and broken down by quintile of family income, are presented in Table IV.1 for the five general population surveys. In addition to the dollar amounts, the table presents the estimated amounts as a percentage of the corresponding amounts for the CPS. While the CPS does not represent the gold standard for estimates of income, and we do not mean to suggest that the CPS estimates are the best, the CPS is the official source of household income and poverty statistics for the U.S., so expressing other survey estimates of income as a percentage of the CPS provides a useful standardization.

Table IV.1

Billions of Dollars
Aggregate Income, All Persons 6,468.4 6,346.3 5,766.2 6,257.7 6,116.2
Family Income Quintile
Lowest 370.5 368.7 391.4 360.0 313.7
Second 774.1 778.4 750.8 808.4 717.7
Third 1,090.2 1,087.4 1,008.8 1,144.7 1,058.4
Fourth 1,446.8 1,415.8 1,307.2 1,461.8 1,420.7
Highest 2,786.7 2,696.0 2,308.0 2,483.0 2,605.8
Sum through Four Quintiles 3,681.7 3,650.3 3,458.2 3,774.7 3,510.4


Percent of CPS
Aggregate Income, All Persons 100 98.1 89.1 96.7 94.6
Family Income Quintile
Lowest 100 99.5 105.6 97.2 84.7
Second 100 100.6 97 104.4 92.7
Third 100 99.7 92.5 105 97.1
Fourth 100 97.9 90.3 101 98.2
Highest 100 96.7 82.8 89.1 93.5
Sum through Four Quintiles 100 99.1 93.9 102.5 95.3

Source:   Mathematica Policy Research, from tabulations of calendar year 2002 income from the 2003 CPS ASEC supplement, the 2001 SIPP panel, the 2002 Full-year Consolidated MEPS-HC, and the 2003 NHIS, and prior 12 months income, inflation-adjusted to calendar year 2002, from the 2002 ACS.

Aggregate income ranges from $5.77 trillion in the SIPP to $6.47 trillion in the CPS—a difference of nearly 11 percent. The other three surveys produce estimates that lie within 2 to 5 percent of the CPS. Aggregate income is $6.35 trillion in the ACS, $6.26 trillion in MEPS, and $6.12 trillion in NHIS. Aggregates in the top quintile may be affected by outliers and by differences in survey practice with respect to the topcoding of public use data, documented in Chapter II. For example, the CPS assigns the means of topcoded values as their respective topcodes, which preserves overall means and totals, but not all surveys do this for all income items. For this reason, we summed the survey aggregates through the bottom four quintiles.25 For every survey, the four-quintile sum is closer to the CPS estimate than is the full aggregate, with the MEPS total exceeding the CPS by 2.5 percent. The SIPP total moves to within 1.5 percent of the NHIS total but is still 6 percent below the CPS.

When we examine the results by quintile of family income, we find that SIPP obtains the most income from the lowest quintile, at 105.6 percent of the CPS total. SIPP’s apparent success in collecting income data from the low end of the income distribution begins to erode noticeably by the second quintile, however. In that quintile, SIPP collects 97 percent as much total income as the CPS. This drops to 92.5 percent by the third quintile, 90.3 percent by the fourth and 82.8 percent in the top quintile. MEPS aggregates exceed the corresponding CPS amounts for quintiles two through four while the ACS aggregates lie within a percent of the CPS aggregates (both above and below) through the first three quintiles before dropping to 98 and 97 percent of the CPS in the fourth and fifth quintiles.

This is only the first of numerous tables, and it examines only one dimension of income, but it presents several striking findings that raise fundamental questions about the collection of income data. One such finding is that with a single question NHIS captures 95 percent as much total income as the CPS, despite the latter’s sizable battery of income questions and its status as the official source of income and poverty estimates for the U.S. Second, with far more income questions than any of the other four surveys, SIPP captures 11 percent less total income than the CPS and 6 percent less than the NHIS’s single question. Third, with its massive sample size and an instrument that is filled out primarily by respondents working without the assistance of a trained interviewer, the ACS nevertheless manages to approximate the CPS more closely than any other survey. Fourth, the MEPS person weights used to prepare the estimates in Table IV.1 were post-stratified to CPS totals by demographic characteristics and the distribution of income relative to poverty. What impact does this have on the MEPS estimates of aggregate income? Would MEPS, with its SIPP-like panel design, yield SIPP-like income estimates in the absence of this post-stratification, or does the use of retrospective annual versus monthly income questions trump the panel design?

More generally, what do these findings say about the collection of income data? Does the strategy of asking respondents about their incomes over the prior calendar year or even the past twelve months have a bigger impact on the amount of income collected than the level of detail that is incorporated into the questions? It will become clear as we progress through this chapter that the limitations of a single-question approach are indeed numerous, but this is a separate issue from the retrospective approach. We also have to ask if the SIPP approach of collecting income at four-month intervals and compiling annual totals month by month is inherently inferior, or whether the other surveys share a common upward bias that arises from their retrospective approach. These are compelling questions, and as we walk through the rest of the findings in this chapter it will become apparent that there are areas in which SIPP clearly excels. Nevertheless, we will also see that outside of these exceptions, SIPP’s estimates of income are consistently low.

View full report


"report.pdf" (pdf, 4.33Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®