The panel finds that each of the major national household survey data sets most suitable for monitoring and evaluation has significant limitations in terms of sample size, nonresponse levels, periodicity, response error, population coverage, or survey content.
Key national-level survey data sets used to monitor low-income and welfare populations are currently not being produced on a timely basis. The value of these data for monitoring low income and welfare populations would be enhanced if they are produced on a more timely basis.
To improve the abilities of national-level survey data sets to measure the effects of changes in broad welfare program components across states, the panel recommends expansions or supplements to the CPS or other surveys.
A fully implemented and continuous American Community Survey has significant potential for use in future welfare policy research. The panel recommends that sufficient funds be devoted to fully implement the survey and that support for the survey at its currently proposed sample sizes is sustained over time.
The potential of the American Community Survey for evaluating welfare policies would grow considerably if the survey included more extensive questions about public assistance benefit and service receipt. The panel recommends adding more detailed questions on public assistance receipt to the survey questionnaire.
The wider array of services provided in social welfare programs and the variation in these programs across states both make measuring program participation and benefit receipt more difficult, especially on a national-level. For national household surveys that measure participation in and benefits received from programs serving the low income population, it is critically important to regularly and frequently review survey questions to keep in step with program and population changes. The panel recommends to the Census Bureau that more resources be devoted towards improving questions on program participation and benefit receipt to better capture program participation. The panel also recommends that HHS work with the Census Bureau to develop mechanisms for regular communication with states to stay abreast of programmatic and implementation changes in the states.
State-level capacity to conduct household surveys of low-income and welfare populations is limited. HHS has begun an important effort to build state capacity for conducting surveys. These efforts need to be continued and expanded.
Administrative data, primarily at the state level, are an important emerging source of information for both monitoring and evaluation. However, there are many significant challenges that prevent them from fulfilling their potential, including the conversion to research use from management use, preservation of data over time, improvements in the quality of individual data items, comparability of data across states, confidentiality and access, and barriers to matching across different administrative and survey data sets. Much more investment in this data resource is needed.
The lack of cross-state comparability is a major barrier to the use of state-level administrative data sets for cross-state monitoring and evaluation.
The panel recommends that HHS, in conjunction with state social service agencies, take steps to further improve the comparability of administrative data across states. These steps should move toward comparable definitions of services and service units and data formats. Building comparability across states will have to be a cooperative effort between the federal government and states and will likely require federal funding of state activities.
The current definition of assistance used to guide state data reporting requirements is very narrow and will not capture many recipients receiving different forms of assistance provided by states. The panel recommends that the Administration for Children and Families consider broadening this definition to include as many types of assistance and services provided as possible.
Administrative data reported by states as part of the TANF reporting requirements will be of limited use for research purposes unless steps are taken to improve them. The usefulness of these data will be improved if the data can be linked to other data sets and if the full universe of cases is reported. The panel recommends that ACF take steps to improve the linkability of these data and encourage states to report the full universe of cases.
Confidentiality, privacy, and access concerns with administrative and survey data and the linking of multiple data sets are important issues, but are currently serving as a barrier to socially important evaluation of welfare reform programs. The importance of access to these data for monitoring and evaluation of programs should be emphasized and efforts to reduce these data access barriers while protecting privacy and maintaining confidentiality should be expanded.
The monitoring and documentation of the actual policies, programs, and implementations of welfare reform at the state and local levels by the federal government has been minimally adequate to date. The panel recommends that the Department of Health and Human Services take active and direct responsibility for documenting and publishing welfare program rules and policies in every state and in every substate area where needed. Continuing updates documenting changes in state and local area rules should also be produced.
In its Annual Report to Congress, ASPE should review current availability and quality of data for welfare reform research, identify high priority data needs, and discuss its own research agenda for data development and technical assistance.