While we were not able to demonstrate the impact of fundamental survey design features discussed in the preceding section, we were able to simulate or otherwise estimate the impact of a number of other survey design features and aspects of post-survey processing. These elements include:
- Family definition, which determines whose income is aggregated and what poverty threshold is used to determine poverty status
- Contemporaneous versus fixed family composition and income for poverty measurement—that is, whether family composition and income reflect changes in composition over the reference period or whether family composition is measured at a fixed point in time and income collected for the members of this fixed family
- Interview month, which affects recall intervals, family composition, the lag between a fixed family composition and the income reference period, response rates, and the quality of income data
- Choice of imputation methodology, including its impact on the distribution of imputed values and their consistency with reported values
- Application of consistency checks between related items collected at different places in the questionnaire
- Application of inflation adjustments when income reference periods differ
- Adding non-periodic withdrawals from retirement accounts to the income concept
- Post-stratification in general and post-stratification on income in particular
Each of these can affect the quality of the income data ultimately released to users from a survey and how the income and poverty data compare to estimates from other surveys.
Several of the surveys included in this study define the family more broadly than the CPS, including unmarried partners and their relatives as well as foster children. Modifying the family definition in this way reduces the estimated number of persons in poverty. Using both MEPS and NHIS, we found that the broader family concept reduced the estimated number of poor by 2.6 million and the poverty rate by 0.9 percentage points. It also changed to some degree the characteristics of the poor. Agencies that adopt a broader family definition for their surveys and analysts who use such data need to be aware that including unmarried partners and their children in the family reduces the number of poor and changes both their demographic composition and the overall picture of family structure as compared to the official measure of poverty.
In both the CPS and most of the other surveys, poverty is measured by summing the annual incomes of people present in the family at the time of the interview and comparing this total family income to a poverty threshold based on the size of the family and its composition. We describe this approach as using a fixed family composition. Simulations with SIPP data indicate that this approach yields higher estimates of poverty relative to an alternative approach that defines family composition and family income contemporaneously—that is, based on who lived with the family each month of the year and how much income they received in each month. Compared to fixing family composition in the final month of the income reference year, the contemporaneous approach reduced the estimated poverty rate by nearly half a percentage point. This result is specific to our simulation but illustrative of the general impact of contemporaneous measurement of income and family composition. The PSID makes use of the contemporaneous approach, and SIPP collects the data needed to do so.
Our simulations also addressed the impact of the length of time between the end of the income reference period and the date when family composition is fixed. The longer the lag, the more opportunity for changes in family composition between the survey date and the income reference year. In our simulation, an interview date three months after the end of the income reference year (as the CPS does) added about a third of a percentage point to the poverty rate relative to defining family composition at the end of the reference year (as MEPS does). Lengthening the time interval raised the estimated poverty rate a modest amount, but its bigger impact was on the number of people who were classified differently relative to no lag.
A surprising result emerged from an examination of allocation rates in the ACS by survey month. Intended to show whether data quality deteriorated over the course of the survey year as the income reference period moved farther away from the previous calendar year, these tabulations showed instead that allocation rates (and non-response rates) for the income questions were higher in March, April, May and June than for other months. In other words, respondents were least likely to respond to the income questions in the months that conventional logic suggested were the best months to ask income questions. The association of high non-response with tax-filing months, and with income levels and income sources usually subject to income taxation, is certainly suggestive but requires further study.
Imputation methods that use respondents as donors will tend to replicate reporting patterns, such as rounding. Allocated income in the CPS, ACS, SIPP, and MEPS shows comparable levels of rounding as reported income. NHIS imputes missing income with a regression model that produces no rounding. PSID includes the imputation of mean values among its allocation methods and shows evidence of very substantial rounding in the allocated values. Clearly, the choice of imputation method has implications for the distribution of imputed values.
The surveys differ in the extent to which they apply consistency checks to related items collected at different points in the survey. Inconsistencies between reported income and reported work activity are notable in MEPS while inconsistencies between the reported receipt of earnings and reported amounts of earnings are observed in NHIS. Inconsistencies such as these present choices to users that will result in different users coming up with different estimates, depending on how they choose to address these inconsistencies. They also provide grounds for critics to question the reliability of any estimates from the survey, even those that may be unaffected by the inconsistencies.
To compensate for the 12 different income reference periods used in an annual ACS, the Census Bureau applies a price adjustment, which converts the reported incomes to constant dollars, using the calendar year in which the survey was conducted as the base. While this achieves a certain uniformity in the income estimates, the approach alters the distribution of income in ways that are inconsistent with actual change over time, as reflected in ACS estimates from consecutive years.
Changes in the way that retirees receive retirement income have been ongoing for decades, yet surveys continue to define and measure retirement income in ways that reflect the earlier world of defined benefit plans providing regular monthly payments. The CPS income definition used in the study excludes non-periodic or lump sum withdrawals from tax-advantaged retirement accounts, which are likely in the long term to substantially replace monthly pension payments based on defined benefit plans. Two surveys—SIPP and MEPS—request lump-sum payments from a range of sources, but they obtain comparatively little additional income with only marginal impacts on elderly poverty. This suggests that considerable work in this area may be needed to develop significant improvements.
Post-stratification is commonly used to correct survey estimates for differential coverage and response rates by demographic groups. While post-stratification in this manner is widely accepted, one drawback is that if the non-responding units within a demographic group are systematically different from the responding units, post-stratification will not take account of this. Instead, the missing units will be given the same distribution of characteristics as the responding units, in effect. MEPS post-stratifies its person-level sample weights to the distribution of poverty status in the CPS. In forcing the sample to fit the CPS income distribution, this may alter the distribution of other characteristics, which may account for some of the ways in which MEPS departs from other surveys—including substantially more earners and substantially more persons living with spouses or living with no relatives.