Income Data for Policy Analysis: A Comparative Assessment of Eight Surveys. Survey Design and Methodology


The single biggest design difference across the eight surveys with respect to income data collection contrasts the subannual approach used in the SIPP and, to a limited degree, the PSID, with the retrospective annual approach used in the other surveys. SIPP collects monthly income data from interviews conducted at four-month intervals. Users of SIPP data may cumulate monthly incomes in any way they wish, and we demonstrate this in constructing estimates of annual income from the SIPP. But the SIPP approach is distinctly different from directly asking people their annual incomes. With the Census Bureau engaged in a redesign of the SIPP that is focused on replacing the current three interviews per year with a single annual interview, the merits of this particular design feature carry significance beyond its methodological interest.

The second biggest design difference among these surveys is the range in the number of questions used to capture total income. Detailed questions on income serve an important purpose beyond whether they lead to better estimates of total income or not, and one would not discard detailed questions from a survey whose major purpose is to capture the breadth and variety of income. But the issue of what level of questioning is needed to capture adequate income is very relevant to surveys that collect policy-relevant or simply analytically important data on other topics but whose users would benefit from the availability of a reasonably good measure of total income.

The use of a rolling versus fixed sample is also a major design difference represented among the eight surveys. A rolling sample consists of non-overlapping subsamples spread across a year (or other time period) and designed to be interviewed sequentially. Within the context of a rolling sample there is an additional difference with respect to the reference period for which annual income data are collected. The ACS uses a rolling reference period, asking respondents their income for the past 12 months while NHIS uses a fixed reference period, asking respondents their income for the previous calendar year. A corollary of the NHIS approach is a variable recall interval, where longer length of recall may bring higher nonresponse and lower-quality data.

Four of the remaining five surveys are longitudinal. They involve repeated interviews with the same individuals over multiple years.1 Attrition and limited representation of additions to the population will tend to make individual panels less representative of the U.S. population over time.2 This is a serious concern with respect to the PSID, which has followed an initial sample of households for 40 years. Another aspect of longitudinal surveys involves the impact of being interviewed multiple times with variations on the same instrument. It is conceivable that the repetition of the income questions—particularly in the SIPP, where the interval between interviews is brief and all of the income questions recur—may improve the quality of the responses over time as respondents learn what to expect. However, our analyses do not explicitly address these features of longitudinal surveys.

View full report


"report.pdf" (pdf, 4.33Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®