Income Data for Policy Analysis: A Comparative Assessment of Eight Surveys. Limitations


While the goal of these efforts was to make the survey estimates of income as comparable as possible, there remain a number of differences due to design or methodological features for which we can make no adjustment. Estimates of the potential impact of some of these differences are presented in Chapter V.

First, our adjustments do not compensate for the fact that the surveys represent populations at different times. The MEPS and SIPP estimates represent populations in December 2002 while the CPS represents a March 2003 population. Both the ACS and NHIS represent an average of populations over a calendar year. The 2002 ACS is weighted to July 1, 2002 while the four segments of the 2003 NHIS are weighted, separately, to February 1, May 1, August 1, and November 1 of that year. Weights for the four segments are combined to create a single annual weight on the public use file, with an effective reference date of mid-June 2003. The PSID interviewed families between March and November 2003, with most of the interviews conducted in April and May. When families were interviewed will determine who was in the universe, but the data were weighted to CPS population controls for March 1, 2003.

Differences in survey timing affect population estimates as follows. If survey B is conducted three months later than survey A, then survey B will exclude people who died in the interval between them, and it will exclude people who, while still living, have left the survey universe as we have defined it by moving outside the country, becoming institutionalized, joining the armed forces, or, if under 15, moved into a household of unrelated persons. Conversely, survey B will include people who have been born or otherwise joined the common universe—for example, by moving (back) into the country, being released from an institution, being discharged from the military, or, for those who were unrelated children under 15 at the time of survey A, have turned 15, been adopted, or otherwise joined a related family.

Second, our adjustments do not compensate for differences in the source of the population controls that were applied or how they were applied, including possible inconsistencies between how the survey post-strata and the post-stratum totals were defined. Some surveys use Census Bureau population estimates directly; others (except for MCBS) use CPS population estimates, or perform their own calculations of control totals based on CPS data. These alternative controls do not agree completely. The different sources of control totals (Census Bureau versus staff or survey contractor calculations) may explain why the MEPS population prior to the application of our adjustments exceeds the December 2002 SIPP population by nearly 500,000 even though MEPS was post-stratified to December 2002 controls.

Third, the adjustments do not correct for the differential treatment of college students living away from home while attending school. These differences affect the size of the ACS population and the composition of ACS, NHIS, and PSID families.

Fourth, the adjustments to the income concept do not address differences on how respondents interpreted what they were to include or not include in the income they reported.

Fifth, the adjustments to family composition involve assumptions about the allocation of family income among the families created by dividing each non-CPS family.

View full report


"report.pdf" (pdf, 4.33Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®