Appropriations conference report language accompanying the targeted welfare outcomes research funding for Fiscal Years 1998 and 1999 included the recommendation that the Department submit its research plan to the National Academy of Sciences (NAS) to receive guidance on research design and recommendations for further research. Accordingly, we provided over $1 million in Fiscal Year 1998, 1999 and 2000 funding to the NAS to convene an expert panel to evaluate current and future welfare reform research.
The Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs released an interim report in September 1999, Evaluating Welfare Reform: A Framework and Review of Current Work, Interim Report, which provided a framework for conducting evaluations of welfare program changes, reviewed current Departmental efforts to evaluate these changes, and provided the Panel's initial conclusions and recommendations. The short-run recommendations for welfare evaluation strategies were consistent with the Department's research on welfare reform, and many of the recommended steps were already being taken.
Findings from the Panel's final report, Evaluating Welfare Reform in an Era of Transition, were released April 5, 2001. The report discussed how best to measure and track program eligibility, participation, child well-being, and other outcomes; data, research designs, and methods for the study of welfare reform outcomes; and needed areas and topics of research. The report also addressed alternative federal and state data sources, the limitations of currently available data, and appropriate evaluation design and methods for analysis. In brief, while the report applauded the Department for a welfare reform research agenda that "is impressive in scope, volume and diversity," it highlighted the need for further improvements and expansions in data collection, development of research questions, and methodological work, as shown in the recommendations below. Highlights of the Panel's recommendations follow.
Broad Data Issues
- The Department should continue and expand its efforts to build capacity for conducting high-quality program evaluations at the state level and for conducting household surveys of low-income and welfare populations.
- National-level surveys (such as the Current Population Survey) should be expanded or supplemented to measure the effects of changes in broad welfare program components across states. More resources should be devoted toward improving national household survey questions on program participation and benefit receipt to better measure program participation and benefit receipt.
- The American Community Survey should be fully implemented and currently proposed sample sizes sustained. More detailed questions on public assistance benefit and service receipt should be added to the survey questionnaire.
- Greater investments need be made to improve the usefulness of state-level administrative data. Steps should be taken by the Department, in conjunction with state social service agencies, to improve the comparability of administrative data across states. The Department should also take steps to improve the linkability of state administrative data and encourage states to report the full universe of cases.
- Expanded efforts should be taken to reduce the confidentiality, privacy, and access barriers to using and linking multiple administrative and survey data sets for welfare program monitoring and evaluations, while still protecting privacy and maintaining confidentiality.
- The federal government should ensure that high-quality and comparable data on human service and social welfare programs and populations are collected for purposes of monitoring the well-being of the low income population and evaluating the effects of welfare reform.
- The Department should identify or create an organizational entity with direct administrative responsibility and authority for carrying out statistical functions and data collection for social welfare programs and the populations they serve. The new statistical agency or unit should also coordinate data collection and analysis activities between states and the federal government.
Program Issues and Research Questions
- The Department should take the lead in defining the important questions to be addressed in welfare reform research and evaluation (with input from states, private foundations, and other stakeholders), and in producing a comprehensive evaluation framework for social welfare programs that considers the major questions of interest and the evaluation methods appropriate for each and guides private and public evaluation efforts.
- The Department should prepare an annual report to Congress, presenting a comprehensive list of the important questions to be addressed in welfare reform research and describing how those questions are being addressed in the overall landscape of welfare reform studies, including whether the appropriate balance of experimental and different nonexperimental methods is being achieved. The report should also review current availability and quality of data for welfare reform research, identify high priority data needs, and discuss the Department's research agenda for data development and technical assistance.
- The Department should consider broadening the current definition of assistance to include as many types of assistance and services provided as possible.
- The Department should take lead responsibility for documenting, publishing, and updating state and substate welfare program rules and policies.
- The Department should sponsor methodological research on nonexperimental evaluation methods, and process and implementation studies. The Panel also recommended more methodological research to assess and improve the credibility of the multiple cohort method (e.g., comparisons of early post-PRWORA leavers to later post-PRWORA leavers, comparisons of post-PRWORA leavers to pre-PRWORA leavers) of evaluating the overall effects of welfare reform.
- More emphasis should be placed on studies that compare current welfare leavers to those who left welfare prior to welfare reform and on studies of divertees, applicants, and nonapplicant eligibles. Further, a welfare dynamics perspective (examining patterns of entry and exit from welfare and the length of spells on welfare) should be incorporated into more welfare reform studies, including leaver studies.
- Experimental methods should be used more in future welfare policy evaluations to evaluate specific reform strategies and different individual broad components of TANF, along with further use of qualitative and ethnographic studies to complement other evaluation methods.
- The Department should take a proactive role in sponsoring experiments at the state and local levels and encourage planned variation and cross-state comparability to yield the maximum general knowledge. The Department also should assume responsibility for synthesizing findings from studies of the consequences of changes in welfare programs.
Copies of the forthcoming report are available through the Academy's website at <www.nas.edu>.
ASPE awarded approximately $807,000 in grants in FY 1999 to support seven researcher-initiated proposals to study important questions related to the outcomes of welfare reform. Through these grants, we supported efforts to analyze a variety of information about low-income individuals (both adults and children) and their families, including their economic and non-economic well-being and their participation in government programs. Final reports have been received from four of the grantees and are summarized below.
RAND Corporation: A Stock-Flow Analysis of the Welfare Caseload: Insights from California's Economic Conditions (1999)
This technical study by researchers at the RAND Corporation examined the methods used in previous studies to explain changes in welfare caseloads in the context of economic expansion and major welfare policy changes. The authors modeled the welfare caseload (stock) as the net outcome of past flows onto and off assistance and explored the implications of such a stock-flow perspective for understanding the determinants of caseload size and its evolution over time. This methodology suggests that the economy has had a greater effect on caseload reduction than suggested by other economic models. Researchers found that approximately half the caseload decline in California could be attributed to the changing economic conditions as measured by the unemployment rate.
Resources for Human Development, Inc.: An Evaluation of Six Welfare-to-Work Programs in Philadelphia (1999)
Resources for Human Development, Inc. (RHD) is a large community agency in Philadelphia that operates six Welfare-to-Work (WtW) programs, which are designed to assist hard-to-serve TANF recipients and their families. The study, completed in November 2000, found that:
- Employed WtW participants reported much higher overall quality of life than unemployed participants.
- Most study members were in poverty or near-poverty before and after WtW participation.
- Child care and transportation support services were heavily utilized by WtW participants.
- WtW interventions were relatively short in duration - 4 to 8 weeks - and most working participants did not perceive the jobs they got as being related to their long-term career goals.
- There appeared to be a significant drop in job retention between the end of training and the 6-month follow-up.
- Tracked families showed few indications of changes in family stability over the period of their WtW involvement.
- WtW participants reported very little involvement with community organizations such as church and school.
SPHERE Institute: Caseload Dynamics and the Business Cycle: Implications for Welfare Policy (1999)
This study built on research the SPHERE Institute conducted under contract with the Public Policy Institute of California to model caseload dynamics using aggregate county caseload counts. It explored the role of economic conditions and caseload characteristics on the role of program performance in California. The author examined cash aid recidivism and the take-up of other forms of assistance by welfare leavers in California. Three separate cohorts of families were tracked - those leaving welfare in 1988, 1993, and 1998 - for a period of 18 months following their exit from the program. Outcomes for the 1998 cohort were compared across different regions of California. Key findings from the report are:
- Recidivism is lower among recent welfare leavers;
- The take-up rate of non-assistance food stamps remains low;
- Enrollment in Medi-Cal (Medicaid) is much higher among recent welfare leavers;
- Welfare recidivism is highest in rural California;
- Non-assistance food stamps take-up rate is highest in rural California.
University of Kentucky Research Foundation: The Impacts and Outcomes of Welfare Reform across Rural and Urban Places in Kentucky (1999)
This study by the University of Kentucky Research Foundation examined the impact of the differential spatial distribution of economic opportunities on the outcomes of current and former AFDC/TANF recipients, including employment and earnings. Findings indicated that patterns of assistance, such as the length of time on TANF and the rate of entry into TANF, reflect urban/rural differences as well as differences across rural areas. Compared to the rest of the state, the rate of caseload decline was lowest in the most remote rural areas.
University of Michigan School of Social Work: Work and Well-Being among Welfare Leavers and Stayers (1999)
This project used data from the Women's Employment Survey (WES) to examine the impacts of welfare reform on economic outcomes as well as on measures of non-economic well-being among specific subgroups of recipients, such as racial minorities and women exposed to domestic violence. WES is a longitudinal data set tracking single mother welfare recipients in an urban Michigan county. Three reports analyzing data from 1997 and 1998 waves of the WES have been released. The first one examines the association of health, mental health, and domestic violence problems with employment, and finds that women who reported depression and serious physical limitations worked less. The second report examines associations between employment and changes in health status, and finds that women who were physically less healthy were also more likely to suffer from major depression. With the exception of a decline in the rate of depression, there was little overall change in the health status of the women during the study period. The third report focuses on domestic violence, and finds that women who had recent and persistent experiences with domestic violence were more likely to suffer from economic hardship than women who had not.
The Urban Institute: Living Arrangements, Work, and Welfare Decisions Among Single Mothers (1999)
This report by the Urban Institute examined the factors affecting single mothers' living arrangements and how these arrangements affect single mothers' decisions regarding work and welfare. The study found that single mothers who live with their parents are less likely to rely on welfare than those who live independently. Living with parents also increased the likelihood that a single mother worked or attended school. When single mothers who live independently are compared with those living with adults other than parents or partners, there is no difference in their welfare and work choices. The authors suggest that policies aimed at encouraging single mothers to live with their parents may reduce welfare use and increase work effort among single mothers.
In fiscal year 1997, ASPE funded the University of Wisconsin Institute for Research on Poverty (IRP) to conduct an administrative data study of the outcomes of families who left AFDC in Wisconsin during 1995. In this 1999 follow-up study, the Institute tracked the outcomes for women in the first study further and the outcomes of a second group of women who left AFDC closer to the time of implementation of Wisconsin Works (known as W-2, Wisconsin's replacement for AFDC, which was implemented in the fourth quarter, 1997). As in the original project, the continuation analysis used linked administrative data from the state including: (1) AFDC data, food stamp data, and Medicaid data from the Client Assistance for Re-Employment and Economic Support administrative database (CARES), and (2) earnings and employment data from the Unemployment Insurance records database (UI).
The final report, Before and After TANF: The Utilization of Noncash Public Benefits by Women Leaving Welfare in Wisconsin, used analyses of linked state administrative data to examine food stamp receipt and Medicaid coverage among mother-headed families who stopped receiving cash welfare assistance for at least two months beginning in the last quarter of 1995 or 1997. The main findings from the study include:
- Over 80 percent of women in both cohorts had at least some earnings during the first quarter after leaving welfare; women in the second cohort, however, were somewhat less likely to be employed in all four quarters after exit compared to women in the earlier cohort.
- Eighty-one percent of those who left in 1997 received food stamps during the first year after leaving, compared to 58 percent of the earlier cohort.
- Ninety-two percent of cases that closed in 1997 received Medicaid within 12 months after exit, compared to 80 percent of those who left in 1995.
- For both cohorts, take-up rates among eligibles for food stamps and Medicaid declined over the 12-month period after welfare exit. While 89 percent of eligibles in the 1997 cohort received Medicaid one quarter after exit, the percentage dropped to 82 percent four quarters after exit.
- Women who worked while still receiving cash welfare were more likely to receive noncash benefits after exit than those not working while on welfare. In addition, women with more months of cash welfare receipt prior to exit were also more likely to receive food stamps and Medicaid after exit.
- Those who left in 1997 faced greater barriers to work (e.g., lower levels of education, more children, very young children) and had substantially lower earnings and income after leaving cash welfare compared to the 1995 cohort.
In addition to pre- and post-TANF cohort comparisons, the study also examined the longer-term outcomes (i.e., three years after exit) of those women who left cash welfare in 1995. The following main points emerge from this analysis:
- Food stamp participation declined over the longer time period. For those who left cash welfare in 1995, 40 percent of those eligible received food stamps in the third year after exit, compared to 60 percent in the first year after exit.
- Medicaid participation also declined over the three-year follow-up for the 1995 cohort. While 80 percent of those eligible received Medicaid in the first year after exit, the percentage dropped to 52 percent in the third year after exit.
The Institute for Research on Poverty (IRP) issued the final report as an IRP Special Report in Spring 2001; further information on the study can be found at <www.ssc.wisc.edu>.
ASPE undertook a data project to create a set of data files to be used for comparing different definitions of the working poor population based on variations in the definitions of worker, family, the poverty threshold, and total income. The project, conducted by Mathematica Policy Research, Inc. (MPR), used data from the Survey of Income and Program Participation (SIPP) that allow analysts to vary the definition of poverty threshold to include several of the alternative definitions for poverty measurement proposed by the National Academy of Sciences(4). ASPE received data reports outlining the composition of the working poor population under different working poor definitions as well as cleaned and documented ASCII (plain text) data files to be used for further analysis and exploration. The resulting data files facilitate comparisons across working poor groups based on multiple definitions and lay the groundwork for future research on alternative poverty definitions and improved understanding of the characteristics of working poor families. MPR will be issuing an additional SIPP data extract file that includes additional poverty measurement variations.
This project estimates the impact of welfare reform on rural and small metropolitan regions since 1993. The report examines 12 rural and small metropolitan regions across the country. It identifies changes in wages and employment for the low-skill labor force over two periods: 1993 to 1996, and 1996 to 1998. The report finds that rural and small metropolitan labor markets easily absorbed welfare recipients who went to work, largely because the economy was experiencing strong growth. The number of new jobs was much greater than the number of welfare recipients who started to work. Wages for low-skill workers declined from 1993 to 1996 in 8 of the 12 regions, but recovered in the second period, from 1996 to 1998. There was little evidence of workers being displaced from their jobs by welfare recipients.
The Federal Parent Locator Service (FPLS) contains the complete national quarterly wage (unemployment insurance and federal employment) and new hire databases as well as registry of child support cases. It provides extensive opportunities for doing welfare and child support research using data from these records. The legislation governing the FPLS stipulates that the data contained in the system must be removed after two (2) years, but allows the creation of research samples which endure past that point. ASPE and ACF retained Social and Scientific Systems to propose design options for a research framework for such samples, which includes matching samples of cases in the system with other administrative data systems in order to get socio-demographic characteristics and program participation data for the samples.
The first step in this project was an analysis of recommended avenues of research using the FPLS. After conducting a series of consultations with prominent researchers around the county, a wide range of research opportunities were identified in a report released in May 2000. Research questions ranged from those that could be answered using only the FPLS database to those that could be answered by linking federal databases such as the TANF database, the Medicaid Statistical Information System (MSIS), or the Child Care Information System to those that required geographic-specific or state policy information in addition to the FPLS.
TANF-related research issues that could be addressed with data from the FPLS or a combined HHS database include characteristics of TANF leavers and TANF entrants over time; participation by TANF leavers in the child support enforcement (IV-D), unemployment insurance (UI), and Medicaid programs; and the relationship between Medicaid receipt, TANF receipt, and earnings.
Some of the child support research issues that could be addressed with data from the FPLS or a combined HHS database include variation in custodial and noncustodial parent employment and earnings based on order and paternity establishment, interaction between the receipt of public assistance (e.g., Medicaid, TANF) and child support order establishment and receipt, speed of order establishment, variation in custodial and noncustodial parent residence over time, and employment and earnings characteristics of noncustodial parents relative to custodial parents over time.
The second part of this project was a report proposing three database design options: 1) a unified welfare and child support research database centered on the MSIS that links in the FPLS and the TANF database; 2) a welfare research database centered around the TANF database, supplemented with FPLS and MSIS data; and 3) a child support research database centered around the Federal Case Registry, enhanced by linking to TANF database or the MSIS. These data provide extensive opportunities to conduct longitudinal welfare and child support operations monitoring, policy development, and research, and to look at the interaction between public programs. For example, an integrated database could tell us about interaction between the receipt of public assistance (e.g., TANF, Medicaid) and child support order establishment. An intra-agency work group has been convened by ACF to assess the proposed options.
Through an interagency agreement with the ACF's Office of Child Support Enforcement (OCSE), ASPE provided funds to support statistical research using matched new hire and quarterly wage data from the files of the National Directory of New Hires (NDNH) database. Use of NDNH data improves the quality of the information on employment outcomes, because this database captures employment in other jurisdictions, and with the federal government, which do not appear in state Unemployment Insurance records. ASPE funds supported programming time and other one-time infrastructure costs related to linkages between the NDNH data and samples drawn for research projects, such as the ASPE-funded grants to study welfare outcomes of former TANF recipients. OCSE performed this match for the District of Columbia, one of ASPE's FY 1998 welfare outcome grantees, and other grantees are considering requesting additional matches. (A link to the District's report, The Status of TANF Leavers in the District of Columbia, Final Report, is available at <http://aspe.hhs.gov/hsp/leavers99/reports.htm#dc.>)
Broad-based population data are essential to studying the well-being of the low-income population. The American Community Survey (ACS) is a new Census Bureau program that will make regular intercensal estimates of the distribution of characteristics of households, families and persons in small areas such as census tracts and for small population groups (for example, specific Asian or Hispanic nationality groups, specific age groups, and so forth). The ACS is currently being conducted in 31 diverse sites across the country. The Census Bureau expects to fully implement the survey in every county starting in 2003. ASPE and the Bureau of the Census jointly sponsored a conference on June 6-7, 2000, which convened a group of researchers, policy makers and local practitioners to explore the potential uses of this new data source and to explore the development of econometric models that combine ACS data with local area administrative data and local business economic data to provide local area data. A conference summary report and the conferees' recommendations are available from the Bureau of the Census.