The assessment presented in this report includes both descriptive and empirical components. The descriptive component, which is presented in Chapter II, provides extensive detail on survey design and methodology as well as on income data and poverty measures for persons and families in each of the eight surveys. The information presented in parallel for the eight surveys includes overall design, timing, recall, reference period, family definition, poverty measurement, content on income and policy-related covariates, income data processing, and public availability and accessibility of income and poverty data. An annotated bibliography of literature relevant to the collection and evaluation of income data was assembled separately from the descriptive component and is presented in Appendix A.
The empirical portion of the report presents findings from comparative tabulations, following a standardized format, that addresses income, poverty, and program participation. These estimates were prepared using the same income measures, definitions and units of analysis for each survey, to the extent that this was possible. Additional findings address methodological issues, specific survey attributes, and questions raised by the detailed information gathered for the descriptive component. These findings focus on the implications of particular design choices.
The empirical analysis does not include any effort to compare the survey estimates with independent benchmarks, which would require a separate study in and of itself. Benchmark construction is difficult because administrative data that are often used to produce benchmarks rarely allow the same degree of flexibility in matching universes and units that we were able to achieve with the survey data alone. Administrative record matches to survey data offer a more promising avenue of research, but they are constrained by legal restrictions on access to administrative data and are very expensive to conduct. A small number of studies using benchmarks or matched survey and administrative records are cited in the annotated bibliography.
Neither do we view any of the surveys as a gold standard against which we can judge the quality of the other surveys. We find it informative to compare the other surveys to the CPS ASEC supplement, given this survey’s status as the official source of income and poverty statistics for the U.S., but such comparisons may be just as informative about the CPS income data as they are instructive about other surveys.
The scope of work for this project specifically excludes recommendations. Rather, the project hopefully provides the material for a separate, independent review that would focus explicitly on recommendations, perhaps including some additional, targeted research as well. The conclusions presented in Chapter VII focus on factual findings and documentation of similarities and differences among the eight surveys.
Finally, a Technical Advisory Group (TAG) representing each survey and the policy research community provided input to the project. TAG members reviewed and commented on drafts of the workplan, the annotated bibliography, the analysis plan, the outline of the final report, the detailed survey descriptions, and the final report. TAG members, Census Bureau staff, and PSID staff at the University of Michigan also provided extensive assistance in obtaining documentation not readily available from published sources or public web sites. In addition, the Census Bureau also performed a major series of tabulations pro bono on the internal files of monthly ACS data. These tabulations provided valuable information that could not be obtained from public use files.