Income is a critical classification variable for policy-related analyses, and together with poverty status is often key in the development of public policy. Most federal household surveys collect some income data and provide measures of poverty status. Yet income is difficult to measure in household surveys, and poverty status depends on how a family is defined, which differs markedly across surveys. Despite many similarities, there are also many differences in the income and poverty concepts used, and different surveys provide markedly differing estimates of income and poverty.
Under contract to the Office of the Assistant Secretary for Planning and Evaluation (ASPE), Department of Health and Human Services (HHS), Mathematica Policy Research, Inc. (MPR) and its subcontractor, Denmead Services & Consulting, have conducted a comprehensive and systematic assessment of the income data and their utility for policy-related analyses in eight major surveys: the Survey of Income and Program Participation (SIPP); the Annual Social and Economic Supplement to the Current Population Survey (CPS); the American Community Survey (ACS); the Household Component of the Medical Expenditure Panel Survey (MEPS); the National Health Interview Survey (NHIS); the Medicare Current Beneficiary Survey Cost and Use files (MCBS); the Health and Retirement Study (HRS); and the Panel Study of Income Dynamics (PSID).
The assessment focuses on three issues:
- The quality and usability of each survey’s income and poverty data for policy-related analyses
- The overall impact of different design and methodological approaches
- Specific design and processing choices that may be related to the quality and utility of income and poverty data in each survey
The assessment is both descriptive and empirical. The lengthy descriptive component provides great detail on survey design and methodology and on income data and poverty measures for persons and families in each survey for the files and years used in the study. It includes overall design, timing, recall, reference period, family definition, poverty measurement, content on income and policy-related covariates, income data processing, and public availability and accessibility of income and poverty data. Additionally, it includes an annotated bibliography of literature relevant to the project.
The empirical portion addresses income, poverty, and program participation using the same income measures, definitions, units of analysis, and time period for each survey, to the extent possible, in standardized tabulations. Additional tabulations address methodological issues, specific survey attributes, and questions raised by the detailed information gathered for the descriptive component.
A Technical Advisory Group (TAG) representing each survey and the policy research community provided input to the project. TAG members reviewed and commented on drafts of the workplan, the annotated bibliography, the analysis plan, the outline of the final report, the detailed survey descriptions, and the final report.
TAG members, Census Bureau staff, and PSID staff at the University of Michigan provided extensive assistance in obtaining documentation not readily available from published sources or public web sites, and the Census Bureau also performed a major series of tabulations pro bono on the internal files of monthly ACS data.
POLICY ANALYSIS CONTEXT
Data requirements for policy analysis are not the same as those for more general research—they are both different and more extensive. Whatever issue is addressed, good income information for policy work is likely to require the following:
- Actual Numbers. Income is often used to determine potential eligibility; benefits or charges may vary with income; and impact at different points in the income distribution is important—policy work needs actual amounts, not broad income intervals
- Comparability with Official Poverty Statistics. Poverty status is important in policy evaluation and in public debate and must be on the same basis as official statistics
- Other Relevant Variables. Work on health usually requires data on health insurance status and utilization, and work on policies concerning the elderly requires data on current retirement contributions and coverage, as examples
- Flexibility on Filing Units.Policy analysis may deal with individuals or part of a family, and may compare different rules for constructing filing units, which requires income data for each person
- Credibility and Reliability. Weaknesses in data underlying policy proposals and cost estimates bring the validity of an initiative into question; significant inconsistencies within a survey or failure to match known population totals lead to challenges to the estimates and proposals themselves
- Transfer or Other Program Participation Data. Efficient policy design requires detail on benefits or insurance coverage already in place, and the administrative systems with which persons already interact
- Immediate Accessability and Speed of Use. Typically the policy process has tight time frames, and unexpected developments when a proposal is being actively considered require new analyses with very quick turnaround times
Accuracy of income data at the lower end of the income distribution is more important than accuracy at the upper end; measures of particular significance include the number and composition of the poor and near-poor, relative importance of key income sources, and insurance status. In addition to income, employment has consistently been an area of policy concern, as a source of self-support and of health insurance coverage, so that accuracy in its measurement is also key. Lastly, randomness as measured by standard errors is not nearly as important as possible bias. The findings of policy analysis, and budget estimates, are presented as point estimates without standard errors, while bias leads to consistent over- or under-estimates.
The surveys differ greatly in overall design and purpose. Five major Federal surveys—SIPP, CPS, ACS, MEPS and NHIS—cover the civilian non-institutionalized population (although ACS excluded group quarters until 2006) but differ in various respects:
- Timing and Reference Period. ACS and NHIS have rolling samples (non-overlapping samples spread across a year), SIPP visits each sample household at strict 4-month intervals, and CPS interviews primarily in March. All but SIPP and ACS get calendar year income; SIPP gets monthly income. ACS gets income for the 12 months prior to the interview; for a given calendar year the ACS income data combine 12 different reference periods.
- Income Detail and Income for Persons. Income detail ranges from the dozens of items collected in SIPP to a single family level variable in NHIS. All but NHIS get income information for every person over 14 years, but NHIS gets only earnings for each person over 17 years and a family income total.
- Family Definition and Poverty Measure. SIPP, CPS, ACS and MEPS have a poverty measure based on the family definition used in official poverty statistics. NHIS uses only a broader definition that treats unmarried partners as married and includes foster children; this affects poverty rates. MEPS provides a second coding of family composition based on this broader definition, which can be used to construct an alternative poverty measure. Due to the difference in reference periods, ACS poverty measures are not comparable to CPS.
- Family Composition Lag. The surveys differ in the timing of family composition used for annual poverty measures. Family composition for poverty estimates is measured December 31 of the income year in MEPS, the month after the income year for ACS, usually March after the income year for CPS, and ranges from January to December after the income year for NHIS. With SIPP, analysts can select the timing used in poverty measures.
None of the other three surveys cover the general population. PSID is a unique survey that has followed the same families and their descendants for 40 years. It has detailed income data that are limited to the head and wife or partner, treats unmarried partners as spouses, has no person totals, and uses a contemporaneous poverty measure. HRS is restricted to persons age 51 or over, treats unmarried partners as spouses, and has detailed income data but no person totals. MCBS covers Medicare enrollees but not their families, asks one income question, and is used primarily to collect information on non-covered services to add to Medicare claims data.
The descriptive component of the study simply required gathering and verifying a great deal of information about each survey and using uniform and consistent terminology to describe key features of the eight surveys. The descriptions apply to the files used in the study and are not necessarily applicable in all detail to other years, since survey content, procedures, sampling, and data may change from year to year. The empirical component was more complex.
The study uses income data for 2002 (HRS and MCBS income for 2003 were deflated with the CPI-U) and applied CPS definitions wherever possible. Survey samples were restricted to approximately the same universe by removing any military and their families, unrelated children under 15, persons institutionalized or deceased by the end of the year, and persons residing outside the fifty States and the District of Columbia. Excluded students were restored to families in PSID. On advice of the TAG, analysis of the MCBS was restricted to the population age 65 or over. In conformity with CPS income definitions, lump sums and irregular payments were removed where included in survey income. However, a number of relatively small differences remain among the surveys in universe, relationship information, income definitions, time lag, and treatment of college students, as well as the larger differences in ACS due to the prior 12 months reference period (rolling reference period) as compared to calendar year in all other surveys and the exclusion of group quarters in 2002.
Work was done on public use files with three exceptions. MCBS has no public use files, but allows protected off-site use with approval and has a standing agreement with ASPE, under which this study operated. NHIS income dollar amounts are available only on an internal file that may not be taken off-site and requires prior approval and usage fees, which the study obtained and paid. ACS interview month is available only on internal Census Bureau files, and the Bureau performed a set of analyses on these files without charge that enabled the study to assess the ACS rolling sample, rolling reference period and price level adjustments, and resulted in other important although serendipitous findings.
Standard Tables. Tabulations were done at the person level, with persons classified by family income using the CPS family definition. A simulation model was built for NHIS to divide family income when CPS families were created from 5.8 million non-CPS families. Sensitivity tests of the model measured the highest and lowest possible impact on poverty rates. A simpler version of the model was used for the PSID, which contains substantial person-level income information, and persons currently living with relatives were included in these families.
Standardized tabulations of persons and family income were performed on each survey by demographic group and income level. Family income was classified by poverty relatives—whether the ratio of family income to poverty thresholds was under 100 percent, 100 to under 200 percent, 200 to under 400 percent, or 400 percent or over—and by family income quintiles—quintiles of persons ranked by family income. Tabulations were repeated for population sub-groups such as persons receiving Supplemental Security Income (SSI), and by health insurance and Medicaid status. Standardized tabulations were also performed for persons with earnings and amount earned, and persons with wages and salaries and wage and salary amounts, reflecting the importance of earned income (82 to 86 percent of total income) and wages and salaries in overall income. Comparison tables were created on other surveys for persons age 51 or over and age 65 or over for comparison to HRS and MCBS, and with demographic and other information restricted to the family head and his wife for comparison to the PSID.
Allocation. Standardized tabulations of persons with income allocations were performed to determine the number of persons and the share of income allocated or imputed, by major income source and family income level. These tabulations were done on each survey containing allocation markers.
Special Analyses. Numerous special tabulations of greater and lesser complexity were performed to address specific methodological issues, including the ACS tabulations described above. The impact of different survey timing of family composition used for annual poverty measures was examined using monthly SIPP data on income and family composition; the use of a single data set ensures that findings are purely methodological and do not reflect differences in data. Comparisons in NHIS and MEPS measured the impact of different family definitions on family and poverty counts. Other special tabulations included the degree of rounding or approximation in income reporting, the impact of including withdrawals from tax-advantaged retirement accounts, and the size and impact of inconsistencies in several surveys where consistency was not ensured by the question sequence or subsequent editing.
There are three groups of study findings: important methodological results that could apply to any survey collecting income data; findings on issues specific to individual surveys; and empirical results of comparisons across surveys.
1. Survey Methodology
Two methodological findings result from analyses that were part of the study design, but the third was an unanticipated result of tabulations examining monthly ACS data.
Within-Year Variations in Response Rates. ACS monthly sample data on over 45,000 households per month show significantly higher non-response and allocation rates for March, April, May and June than for other months.
- Allocations rose from 19.0 percent of total income in February to 22.8 percent in March and 24.6 percent in April, and all differentials were highly significant
- Differentials for these months were found for five of seven income sources and were statistically significant for wages and salaries, Social Security, asset income, and pensions, although not for self-employment
- The elevation in non-response rates did not occur for SSI or public assistance
- The differentials for March, April, May and June were statistically significant for all quintiles and for all income subgroups above the poverty level
- The amounts by which non-response rates rise during these months increased with income, although average non-response rates decreased with income
The strong pattern in income non-response has implications for overall survey design. The association with tax-filing months and with income levels and income sources usually subject to income taxation is certainly suggestive but requires further study.
Dynamics of Family Composition. Measuring family size and composition at different points in time to calculate poverty rates from the same income for the same year, in SIPP longitudinal data, shows that poverty rates rise as the time increases between measurement of income and measurement of family size and composition.
- Poverty rates are lowest when income and family composition are measured at the same time, or contemporaneously, in monthly data
- As the interval increases between the income reference period and the fixed date at which family size and composition are determined, the number of persons incorrectly classified as poor increases faster than the number of persons incorrectly classified as not poor, and poverty estimates are mildly biased upwards
- The total number of persons incorrectly classified either as poor or not poor greatly exceeds the net change in the number classified as poor and the poverty estimate
- An average of the poverty calculations for each of the 12 months of the next year (NHIS) will yield more poor than calculations based on the next March (CPS), and both will be higher than calculations based on December 31 (MEPS)
- Larger differences are found for minorities, single parents with children, welfare and Food Stamps recipients, and Medicaid enrollees, as the time lag increases
This finding is purely methodological and is based on SIPP data with very detailed income information, a maximum recall of five months, and an average recall of three months. With this data, the poverty rate based on a March family was 0.6 percentage points above a contemporaneous measure, and the poverty rate based on a December (of next year) family was 0.6 percentage points above the rate based on a January (of next year) family. Other surveys have less or no income detail compared to SIPP, and have long recall intervals that average 12 ½ to 18 months and can be as much as 23 months. In surveys with less income detail and longer recall intervals the impact could well be larger, and standardized tabulations cannot adjust for these differences.
Family Definition. Poverty calculations with NHIS and MEPS data show that a broad family definition—including unmarried partners and their relatives in families—reduces the number of poor compared to the conventional family definition in CPS. The different definitions also give different pictures of family arrangements. MEPS provides both family definitions and reports income at the person level, so family income and poverty can be constructed for either definition. NHIS codes only the broad definition and reports a single family income total, so the study simulated CPS families for 17 million people.
- In both NHIS and MEPS, when we used the broad or NHIS family definition to calculate poverty rates the number of poor declined by 2.6 million and the overall poverty rate by 0.9 percentage points—the estimated declines in NHIS are plus or minus 230,000 persons, or less than one-tenth of a percentage point
- In MEPS the poverty rate for children declined by 1.7 percentage points, and the poverty rates for single parents and their children declined by well over five percentage points each under the NHIS family definition
- Poverty rates for the elderly were unchanged when the definitions were compared
- In both NHIS and MEPS, quintile bounds all shifted upwards by $1,000 to $2,000
A number of surveys use broader family definitions treating unmarried partners as families. Broader definitions reduce both the number and demographic composition of the poor and change the overall picture of family structure.
2. Survey-Specific Issues
Many issues or procedures are unique to one or two surveys, and one purpose of the study was to identify and describe such issues, and measure their impact if possible.
Design Features. A few design features can be examined empirically, but most can only be described as a context for interpreting the results of standardized tabulations.
- MEPS is designed to piggyback on the NHIS sample, sampling from successful NHIS interviews; only persons selected from NHIS and those who later join MEPS families but were not in scope for the NHIS sample are assigned person weights.
- MEPS respondents who are not eligible for person weights may be eligible for family weights, but not everyone who receives a person weight receives a family weight. This means that the samples for person-level and family-level analysis do not overlap completely. Specifically, 10.4 million persons (weighted) with CPS family weights and 13.0 million with MEPS family weights have no person weights, and 6.1 million persons with person weights but one or more non-interviewed family members have no family weights. This design feature is unique to MEPS among the eight surveys.
- MEPS adjusts (post-stratifies) person weights to ensure that the MEPS public use file yields the same poverty rates by demographic groups as the CPS; MEPS also adjusts (post-stratifies) family weights to ensure that the MEPS public use file yields the same counts by family size and family type as the CPS.
- ACS income data combine 12 reference periods for a given year that on average lag the calendar year by six months; income is adjusted to the calendar year level for inflation but cannot be adjusted for productivity, unemployment or other factors, nor will it fully reflect economic shocks during the year such as sharp changes in energy costs, food prices, or credit availability.
- The ACS rolling sample, rolling reference period and inflation adjustments were examined through tabulations for each separate month, with and without inflation adjustments, and across income levels, but no discernable patterns were found.
- PSID is a panel survey following the same families and their descendants for 40 years, designed for longitudinal rather than cross-sectional work; responding families may no longer be representative and weighting is done at a family rather than person level.
- Preliminary PSID weights use CPS counts of primary families and primary individuals as control totals, excluding unrelated subfamilies and secondary individuals, and do not fully reflect definitional and universe differences between PSID and CPS. PSID weights to 261.5 million persons, compared to 282.6 million in CPS; excluded groups account for 8.1 million of the 21.1 million person difference.
Editing and Consistency. Income data processing typically includes overall consistency checks, such as whether workers have earnings, those with earnings report working, or whether the type of employment—working for others or self-employment—matches the type of earnings reported. MEPS collects employment and dollars of earnings in separate sections of the instrument (and collects the employment data three times per year but dollars of earnings only once a year). In order to maintain the independent information provided by the responses, which sometimes disagree, MEPS does not impose consistency edits. Here and elsewhere, where edits were not made, the study measured the impact.
- In NHIS, 4.3 million persons reported receiving wage and salary or self-employment income for the year but have no work activity or amounts earned in the same year, and another 4.0 million persons reported working, and amounts earned, but no receipt of wage and salary or self-employment income for the same time period.
- In MEPS, 6.6 million persons reported wage and salary or self-employment income for the year but no work activity on the detailed JOBS file of employment for the same time period.
- In MEPS, 2.6 million persons reported details of one or more jobs working for others or themselves during the year but no wage and salary or self-employment income for the same time period.
- In MEPS, 16.5 million persons with only self-employment for the year on the detailed JOBS file reported $620.2 billion of wages and salaries for the same time period. Re-classifying the entire amount as self-employment income would give MEPS more than any other survey whereas MEPS shows little self-employment income otherwise.
- SIPP skipped around questions on net profits for 2.0 million self-employed in sole-proprietorships and some partnerships when no monthly draw was reported; this omission of some self-employment income was corrected in the 2004 panel.
- SIPP does not edit or impute monthly work activity against monthly earnings or monthly earnings against monthly work activity, yet finds less than one-half million persons with either work activity but no earnings or earnings but no work activity on an annual basis, compared to 8.3 million in NHIS and 9.2 million in MEPS.
NHIS Family Income Consistency. Most household surveys don’t require consistency checks on family income, since it is a calculated sum of income across sources and across persons. NHIS gets family income, and earnings (never negative) for persons, but does not determine whether total earnings in a family exceed the family’s income.
- For 61.7 million persons and 9.9 million poor, family earnings exceed family income; family earnings are over $10,000 above family income for 27.6 million people and over $20,000 higher for 15.4 million, with the excess totaling about $290 billion.
- Using higher family earnings to determine poverty reduces the poverty rate 1.4 percentage points on either the CPS or NHIS family definition, and the number of poor by 3.9 or 4.0 million for the CPS and NHIS family definitions, respectively.
- Using higher family earnings improves poverty status for another 12.3 million by shifting them from 100 to 200 percent of poverty to above 200 percent of poverty, or from 200 to 400 percent of poverty to above 400 percent of poverty.
- Earnings and/or family income were imputed for most NHIS families with total earnings in excess of total income; they were imputed for 71 percent of all persons with family earnings greater than family income, 83 percent of those whose poverty status changes and 88 percent of those with a difference of more than $20,000.
These excess earnings were excluded when non-CPS families were split to meet CPS family definitions for the study’s standardized tabulations. Instead, the combined income of split-off CPS families was constrained to equal the income of the original NHIS family for which the data had been collected.
Income Definition. The CPS income definition used in the study excludes non-periodic or lump sum withdrawals from tax-advantaged retirement accounts, that are likely in the long term to substantially replace pension income based on defined benefit plans. Tabulations to assess the impact of these withdrawals were done in SIPP and MEPS; other differences remain that cannot be assessed.
- Standard tabulations included $3.3 billion of periodic IRA, Keogh or 401(k) payments in CPS and $18.7 billion in SIPP; non-periodic withdrawals of $12.7 billion were restored to income in SIPP but had no significant impacts
- Taxable IRA withdrawals of $65.6 billion were restored to income in MEPS and reduced the overall poverty rate by 0.1 percentage points and the poverty rate for the elderly by 0.5 percentage points
- MEPS uses Internal Revenue Service definitions that exclude contributions to tax-deferred retirement accounts such as 401(k)s from wages, treat income from self-employment other than a sole proprietorship or farm as rents, royalties or estate income, and exclude interest and dividends from tax exempt municipals—these definitional differences cannot be removed and their impact cannot be measured
- None of the surveys collect information on defined contribution retirement benefits comparable to data on income from traditional pension plans
Relationship Detail. Surveys differ in the information collected on relationships within households or families, whether to the reference person or among other household or family members; this may limit information on family structure and reduce flexibility in constructing potential filing units. Surveys also differ in treatment of college students.
- ACS has no information on relationships among persons not related to the household reference person, so that unrelated subfamilies cannot be identified and their members are treated as unrelated individuals; treating the 1.2 million persons in unrelated subfamilies in CPS as unrelated individuals reduces the number of poor by 173,000 and excludes almost 220,000 poor children under 15 from the poverty universe
- SIPP and CPS only identify parental or marital relationships among persons not related to the household reference person, so that only husband-wife and parent-child unrelated subfamilies can be identified, not other related subfamilies, e.g., siblings
- MEPS identifies members and the reference person of CPS-defined families, and while relationships are coded only relative to the MEPS family reference person, there are virtually no cases where the relationship to the CPS family reference person cannot be discerned
- MEPS sample members with person weights but no family weights have family members who are not on the public use file; these sample members represent 6.1 million persons in families of “undefined size”; 2.4 million are in families with no reference person on the public use file
- Persons in MEPS families of “undefined size” have a poverty rate of 34.5 percent and are disproportionately minority, female, children, and single-parents, but less likely than average to be uninsured, on Medicaid, or on welfare or Food Stamps
- SIPP, CPS and MEPS include college students in the parental family and CPS does not interview in dormitories; NHIS and ACS include students where they currently reside, so those in student housing in the interview month in NHIS become single individuals and in ACS are omitted until 2006; and HRS and PSID treat students away from home as “institutionalized”
- ACS excludes group quarters until 2006; group quarters in CPS have 205,000 residents of whom 115,000 are poor, but CPS includes over two million residents of college or university housing in parental families that the ACS includes in group quarters; for 2006 and later, students living in dormitories are excluded from the ACS poverty universe, but if included could increase ACS poverty rates up to 0.7 percentage points
- PSID retains separate family status for persons—usually grown children or aging parents—previously living on their own but currently living with a related family
Availability and Utility. Most of the surveys have public use files with dollar amounts for income by source for a month or year for every person above some age. The absence of any of these attributes compromises the usefulness of survey income information for policy work.
- NHIS has no actual dollar amounts on public use files, and MCBS has no public use files; MCBS files are available for off-site use with appropriate confidentiality protections but NHIS files with dollar amounts may not be taken off-site, and users obtain and retain only tabular or analytic output
- ACS income data on public use files (which are samples of the internal files) have neither the month of data collection nor month-specific inflation adjustments; an average of the 12 monthly adjustment factors is provided on the public use file but it under-adjusts months early in the year and over-adjusts months later in the year
- NHIS has no person-level income totals and gets family income only on the NHIS family definition, which is not comparable to official statistics; it required complex modeling to create CPS families, and income estimates for any other filing units would be problematic, especially without files available for off-site work
- PSID has a great deal of income detail for the family head and spouse (or partner) but has no income totals for persons nor income by source for other family members
- ACS income amounts on public use files have been rounded (after top-coding) with items below $1,000 rounded to the nearest $10, those from $1,000 to $50,000 rounded to the nearest $100, and above $50,000 rounded to the nearest $1,000
Comparisons Across Surveys
Empirical findings using CPS income and family definitions show major differences among the eight surveys, including varying measures of total income, the distribution of income, earnings and earners, number and demographic composition of poor, poverty rates, program participation, uninsured and low-income uninsured. Additional findings on response rates, allocation and imputation rates and rounding provide information on the quality and reliability of income data. However, standardization cannot adjust for many design features, including the ACS reference period, post-stratification in MEPS, ACS lack of group quarters in 2002, significantly lower population totals in PSID, person-level income data restricted to the family head and wife in PSID, and the contemporaneous poverty measure embedded in PSID. Other survey differences relate to unrelated subfamilies, timing of family composition, treatment of students, and differences in defining income. Most empirical comparisons involve the five large general population surveys and PSID, although the small PSID sample prevents reliable comparisons for small sub-populations.
Total Income and Income Distribution. The largest difference among surveys is a lower total or aggregate income in SIPP, affecting the upper part of the income distribution. Administrative data matches have shown the difference is not due to an underrepresentation of higher-income families in SIPP, and it is possible that the lower SIPP estimates are an artifact of monthly income reporting and shorter recall intervals.
- Excluding PSID, aggregate income ranges from $5.77 trillion in SIPP to $6.47 trillion in CPS, a difference of $702 billion and over 10 percent; the difference is more than accounted for by $884 billion less wages and salaries in SIPP compared to CPS
- Aggregate income is $6.35 trillion in ACS, $6.26 trillion in MEPS, and $6.12 trillion in NHIS; NHIS is $6.41 trillion if earnings are used for families whose earnings exceed income
- PSID, despite a weighted population of 21 million fewer persons than CPS, has the highest aggregate income at $6.72 trillion
- SIPP has the least inequality in income distribution, and NHIS the most, with ACS and PSID close to CPS; NHIS is also close to CPS if earnings are used for families whose earnings exceed income
Earnings and Earners. In all surveys, earnings (wages and salaries plus self-employment income) account for 82 to 86 percent of aggregate income. Numbers of earners and average earnings differ somewhat among surveys but differences among numbers of self-employed or working for others and among amounts earned from wages and salaries and self-employment are much larger.
- Number of earners ranges from 147.4 million in NHIS to 160.4 million in MEPS, with 151.9 million in ACS, 150.4 million in CPS and 154.1 million in SIPP
- Average earnings per worker vary from $30,899 in SIPP and $32,813 in MEPS to $35,707 in NHIS and $35,591 in CPS; ACS is $34,279
- If those reporting work activity in MEPS or receipt of earned income in NHIS, and those skipped around self-employment income questions in SIPP are included, the range on number of earners changes to 150.4 million in CPS to 163.0 million in MEPS, with 151.7 million in NHIS and 156.0 million in SIPP; ACS does not change
- Number of wage and salary workers, reported for the three Census Bureau surveys, has a narrow range, from 140.4 million in SIPP to 142.4 million in ACS; however, SIPP finds more self-employed than either of the other surveys
- Average wages and salaries per worker are lowest in SIPP at $29,514 and highest in CPS at $35, 514, with ACS mid-way between
- PSID gets earnings only for the family head and wife; comparisons with similarly restricted counts in CPS, SIPP and MEPS find higher proportion of earners and higher average earnings in PSID than the other surveys
- Comparisons between PSID and other surveys for wages and salaries follow the same pattern—PSID has the highest proportions of wages and salary workers and higher average wages and salaries per worker than the other surveys
Number of Poor and Poverty Rates. Standardized comparisons of poor and poverty rates show a wide range. Measures for ACS are affected by its lack of group quarters and treatment of unrelated subfamilies, but these factors may have offset each other.
- Total poor and poverty rates (excluding the contemporaneous PSID measure) vary from 33.2 million and 11.8 percent in SIPP to 41.6 million and 14.7 percent in NHIS—a range of 8.4 million people and 2.9 percentage points
- CPS, ACS and MEPS poverty counts and rates are similar to each other, at 34.4 million and 12.2 percent in CPS, 34.6 million and 12.5 percent in ACS, and 35.3 million and 12.5 percent in MEPS; MEPS is post-stratified to match CPS but adjustments for comparability produced differences
- Poverty rates in PSID are even lower than those in SIPP when both are measured on the same contemporaneous basis—9.8 percent compared to 10.6 percent for all ages—and are also lower for age 65 or over, children, whites and blacks
- SIPP finds fewer poor age 65 or over than the other surveys except PSID, and more poor children than other surveys except NHIS; NHIS has 2.3 million more poor children than CPS, 1.4 million of them living in husband-wife families
- Total numbers and percentages below 200 percent of poverty range from 83.9 million and 30.2 percent in ACS to 95.5 million and 33.7 percent in NHIS—a range of 11.6 million persons and 3.5 percentage points
- CPS and MEPS counts and rates of those below 200 percent of poverty are similar to each other, at 86.2 million and 30.5 percent in CPS and 87.5 million and 30.9 percent in ACS; SIPP is somewhat higher at 89.5 million and 31.8 percent
- The rates below 200 percent of poverty in PSID are also lower than those in SIPP measured on the same basis, 25.5 percent for all ages compared to 29.9 percent
Program Participation. Counts of persons with SSI, welfare, on Medicaid, or living in a family receiving welfare and/or Food Stamps vary sharply among surveys, sometimes by a ratio of two to one. Generally, SIPP has the highest levels of program participation, and CPS and PSID frequently have the lowest.
- SIPP finds 3.4 million persons who ever received welfare during the year, compared to 2.9 million in ACS, 2.2 million in CPS and 1.8 million in MEPS
- SIPP finds 8.4 million persons who ever received SSI during the year, compared to 6.4 million in MEPS, 5.5 million in NHIS, 4.9 million in CPS and 4.5 million in ACS
- SIPP finds 31.4 million persons in families receiving welfare and/or Foods Stamps during the year, compared to 24.3 million in ACS, 22.0 million in NHIS, 20.5 million in CPS and 20.2 million in MEPS
- PSID measures receipt of SSI, welfare or Food Stamps only for the family head and wife; comparisons with similarly restricted counts finds 0.9 percent of persons received SSI during the year in PSID, CPS and ACS, and 1.6 percent of persons in SIPP
- PSID and the comparable count in CPS find 7.3 percent of persons living in families whose head or wife received welfare or Food Stamps during the year, and comparable counts find 8.8 percent in ACS and 11.2 percent in SIPP
- SIPP finds 48.1 million persons ever enrolled in Medicaid during the year, compared to 41.2 million in MEPS and 32.9 million in CPS; PSID has little more than half the number in CPS
- MEPS finds 35.0 million persons currently enrolled in Medicaid, compared to 33.3 million in SIPP and 29.9 million in NHIS
Uninsured. Five surveys contain information on who had health insurance coverage during the last year, and for these surveys the uninsured are persons never covered during the year. Three surveys have information on who is currently uninsured. Counts of uninsured differ greatly, in part because uninsured are a residual after positive responses on health coverage, so that low measures of e.g. Medicaid participation can translate into high counts of uninsured.
- CPS finds the highest level of uninsured last calendar year at 41.8 million persons, compared to 33.3 million in MEPS, 27.5 million in NHIS and 22.9 million in SIPP
- PSID, with 21 million fewer persons than CPS, finds 35.5 million persons uninsured last calendar year or 13.6 percent—slightly below the 14.8 percent in CPS but higher than 11.8 percent in MEPS, 9.7 percent in NHIS and 8.2 percent in SIPP
- Uninsured last calendar year with income under 200 percent of poverty range from 22.9 million in CPS, through 18.2 million in MEPS and 17.8 million in NHIS, to 14.2 million in SIPP; PSID has fewer low income persons but finds 22.9 million are uninsured
- Uninsured children last calendar year under 200 percent of poverty range from 5.2 million in CPS and 4.6 million in PSID to 2.6 to 2.9 million in MEPS, NHIS and SIPP
- Counts of persons currently uninsured, a measure not contained in CPS, are much closer—47.5 million in MEPS, 42.9 million in SIPP and 41.3 million in NHIS
- The ratio of current uninsured to never insured last calendar year in the two surveys with both measures is 1.87 in SIPP and 1.42 in MEPS; the ratio is a measure of turnover and a proxy for duration of uninsurance—higher ratios indicate shorter spells of uninsurance
- Counts of currently uninsured below 200 percent of poverty are very close, and number 25.1 million in NHIS, 24.9 million in SIPP and 24.7 million in MEPS; children account for 6.7 million of these in SIPP, 4.8 million in NHIS and 4.7 million in MEPS
- For the uninsured below 200 percent of poverty, turnover rates are lower, suggesting longer spells of uninsurance—the ratio of current uninsured to never insured last calendar year is 1.76 in SIPP and 1.36 in MEPS
Restricted Populations. Two of the surveys cover subsets of the general population—persons age 51 or over, and Medicare enrollees—with limited information and significant differences from other surveys. Tabulations of income and demographics were done on major surveys as comparably as possible for comparison, using the RAND file for HRS.
- Comparisons of persons 51 or over in CPS, SIPP and ACS with the same population in HRS found those in HRS a little more likely to be living with other relatives and less likely to be living alone; comparisons also found higher family incomes in HRS than for comparable persons in CPS, SIPP and ACS, with HRS incomes 20 to 30 percent higher than CPS and SIPP and about 15 percent higher than ACS.
- Comparisons of persons 65 or over in CPS, SIPP and ACS with Medicare enrollees 65 or over in MCBS found little or no differences in living arrangements but substantially more income, $940 billion for 32.0 million persons 65 or over in MCBS compared to $683 billion for 34.0 million elderly in SIPP, $730 billion for 34.2 million elderly in CPS, and $796 billion for 33.6 million elderly in ACS.
- In CPS, SIPP and ACS, average income per person 65 or over living with a spouse is very similar to that of elderly living alone; in MCBS, average income of enrollees living with a spouse is almost double that of enrollees living alone. The MCBS gets income of the enrollee and spouse for married sample persons, although the MCBS sample frame consists of individual enrollees; income of spouses also enrolled in Medicare is represented by other sample persons and is thus double-counted.
Non-Response and Item Non-Response. Non-response in household surveys is a serious issue. High initial rates of refusal (survey non-response) may lead to non-response bias; longitudinal attrition is a lesser issue given the availability of data from earlier interviews. Replacing missing income information (item non-response) through allocation introduces a stochastic element. In addition, methods vary and may also lead to bias. Both the variability and potential bias are reduced when allocations incorporate partial information supplied by respondents, such as bracketed amounts (collected from respondents who would not provide dollar amounts), wage rates and hour worked, and, for panel surveys, amounts reported in earlier waves. We include as allocations our own pro-rating of part-year income in SIPP to create an annual amount. Allocation rates could not be computed for MCBS or HRS but are reported for the other surveys.
- Initial response rates range from over 97 percent for ACS, the only mandatory survey, to 70 percent for MEPS; SIPP and NHIS are 88 and 89 percent, and CPS is 92 percent for the underlying monthly survey, but about 11 percent of persons with income in CPS are whole imputes who have refused to answer the ASEC supplement; the initial response in 1967 for the major component of the PSID sample was 79 percent.
- Allocation rates range from 17.6 percent of total income in ACS to 42.7 percent in MEPS; SIPP, CPS and NHIS have similar rates from 32.4 to 34.2 percent, including whole-person imputes in CPS and pro-rated income for persons present only part of the year in SIPP.
- When allocations based on partial information supplied by the respondent are excluded, allocation rates range from 6.9 percent of total income in SIPP and 7.1 percent in MEPS to 30.2 percent in NHIS. Allocations in the CPS and ACS do not make use of partial information (as defined here).
- In the five major surveys, allocation rates (as percentages of income from that source) are highest for asset and self-employment income; other income sources may have high allocation rates in one survey but not another.
- Nonetheless, allocated earnings account for 77 to 85 percent of allocated income in the major surveys, and allocations of income from other sources range from minimal to less than ten percent of all allocated income in any survey.
- As shares of total income, allocated earnings (with or without partial information) range from 14.5 percent in ACS to 36.4 percent of income in MEPS; in SIPP, CPS and NHIS allocated earnings have similar shares of 25 to 27 percent of total income.
Rounding. Round numbers suggest inexact reporting or approximations, but the percent of persons with income amounts exactly divisible by $5,000 or $10,000 varies with the number of questions, type of income, and allocation method. If many income amounts are summed, rounded totals are less likely, and hot-deck but not regression-based allocations carry rounding over from donor records. The rounding tests were restricted to amounts below $52,500.
- In SIPP, with detailed income questions and monthly data, virtually no one has rounded income amounts, whether reported or allocated
- In NHIS, with single annual amounts, 40 percent of earners and 36 percent of families report amounts divisible by $5,000, and 23 percent of earners and 21 percent of families report amounts divisible by $10,000; no rounding is found in allocations, which are regression-based
- In CPS and ACS, 28 to 30 percent of earners report amounts divisible by $5,000, and 16 to 17 percent report amounts divisible by $10,000; allocations have similar levels of rounding in CPS but are one-third lower in ACS
- PSID and MEPS have less rounding—19 to 23 percent of earners report amounts divisible by $5,000, and 10 to 12 percent report amounts divisible by $10,000; in PSID allocations are higher but in MEPS allocations are one-third lower
- In contrast to earnings, Social Security and retirement income have little rounding—less than 10 percent of recipients of either reported amounts divisible by $5,000 in CPS, SIPP, ACS or MEPS
- PSID has almost no rounding of family Social Security or transfer income of the head and wife—less than 5 percent of families reported amounts divisible by $5,000
Many of the study findings address ways in which survey design and methodology impact the utility of survey income data for policy analysis, although some findings suggest simple and feasible improvements. It is clear that the quality of income data varies substantially. In large part this is a reflection of the different purposes of the various surveys. But we also find that design features adopted to enhance the quality of income data do not always work as intended.
It was not within the scope of this study to make recommendations. However, the study provides the groundwork for both a discussion of future directions and work on issues in individual surveys and, hopefully, will be a solid starting place and perhaps the basis for recommendations on survey improvements and future innovations.