 |
Office of the |
Assistant Secretary for Planning & Evaluation |
|
|
|
|
HUMAN SERVICES POLICY |
[ Report’s Main Page | Introduction & Overview | 1999 Projects | 1998 Projects | Appendix A | Appendix B ]
In its report for the FY 1999 Appropriation for the Department of Health and Human Services, the Conference Committee added $5 million to the Policy Research account in the Office of the Secretary and directed in its report that the funding was to study the outcomes of welfare reform:
“The conference agreement includes $5,000,000 for a study on the outcomes of welfare reform. The conferees recommend that this study involve statespecific surveys and data sets, survey data on the impacts of state waiver programs, and administrative data such as Food Stamp, Social Security and Internal Revenue Service records. The study should measure outcomes in both low and high economic growth areas of the country. The conferees strongly urge the Department to submit its research plan to the National Academy of Sciences to provide guidance on research design and recommend further research. The conferees further expect an interim report to be submitted to the Appropriations Committees within six months.” (H. Rept. 105-825, page 1291)
The following report has been prepared by the Office of the Secretary, Assistant Secretary for Planning and Evaluation (ASPE), in response to the requirement for an interim report. Except where otherwise noted, this report discusses only the welfare outcomes research agenda supported by the $5 million targeted appropriation. It does not attempt to address ASPE’s or the Department’s overall welfare research agenda, except to the extent that some projects were supported jointly by welfare outcomes funding and funds from other sources.
Passage of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) in August 1996 was a watershed event in the devolution of responsibility for social programs from the federal government to the states. As such, interest has been and remains high about the effects of welfare reform, and questions abound about what is known about the implementation of welfare reform, including state policy and spending choices. The Department acted early to create a research, evaluation and data strategy that would assure that the implementation of welfare reform and its effects would be documented. The infusion of Policy Research funding dedicated to studying welfare outcomes has been invaluable to our efforts to add to and enhance the information about welfare reform outcomes that will be available to the Congress and other interested parties.
There is a broad array of ongoing research about welfare reform being funded by the Department and other public and private funders. We have used the targeted Policy Research funds to fully fund some projects, to fund specific portions of some larger studies, and to co-fund with other federal and state agencies yet other projects. As a result our research, evaluation and data activities cover a wide spectrum of welfare outcomes policy interests.
Despite the breadth and scope of these efforts, from a research perspective it is still early and our knowledge is still quite limited in many areas.
In addition, there are many factors that limit what research can accomplish, including a wide variation in what is happening across states, between local sites, and even from workertoworker; the continuing evolution of state policies and devolution of state responsibilities; the difficulty in tracking people who may no longer be receiving any public benefits; and the confounding effects of other variables (such as the economy) in changing outcomes. Because of these factors, the ability of research, evaluation, and data to completely answer questions is always limited.
To optimize the potential that these targeted funds will increase the Department’s understanding of the outcomes of welfare reform, the Office of the Assistant Secretary for Planning and Evaluation (ASPE) has created, often with other funding partners involved, a portfolio of studies and strategies. Careful attention has been paid to identifying on-going research, evaluation, and data activities which could be enhanced or modified, and to identifying activities being funded or planned by other entities that would help fill the knowledge gaps or provide jointfunding opportunities while avoiding unnecessary duplication. In keeping with the recommendations of the conferees, our plans include studies involving statespecific surveys and data sets and utilizing administrative data from other programs such as Food Stamps and Social Security records. In selecting our state and county grantees, in particular, ASPE has sought to measure outcomes across varied dimensions, including low and high economic growth areas of the country. ASPE has also provided funding to the National Academy of Sciences’ Committee on National Statistics to convene a panel of experts to review data needs and methods for evaluating the outcomes of welfare reform. Our specific activities and plans in each of the recommended areas are summarized below.
The largest portion of our FY 1998 welfare outcomes funding went to partnering with states and counties through grants to study the experiences of people who left the TANF program (“leavers”). Similarly, our plans for FY 1999 call for the largest portion of welfare outcomes funding to go to new state/county grants, with an emphasis on families who have been diverted from welfare, and to continue support for selected FY 1998 grants.
Four of the FY 1998 ASPEfunded grantees — Arizona, Washington, San Mateo County in California, and Cuyahoga County in Ohio — have released interim reports that used linked administrative data sets to track families who left welfare in the second half of 1996. These interim reports provide interesting preliminary findings about former TANF recipients in the areas of employment, returns to TANF, and participation in other programs. Other grantees are expected to release interim reports within the next few months; and more detailed findings, including information gathered through survey data, will be presented in the final reports, forthcoming in late 1999 and early 2000.
Although it is difficult to compare findings across studies, comparisons among ASPEfunded studies are facilitated by the adoption of a common definition of the “leaver” study population as “all cases that leave cash assistance for at least two months.” (This definition excludes cases that re-open within one or two months, because such cases are more related to administrative “churning” than to true exits from welfare). Moreover, these four studies focus on using administrative data to track singleparent families leaving TANF in a similar time period — the third or fourth quarter of 1996(1).
The studies differ, however, in important areas of research methodology, such as how to operationalize some of the outcome measures (for example, three studies count anyone with earnings as employed, while the fourth — Cuyahoga County — counts as employed only those with earnings of $100 or more per quarter). Furthermore, the states differ in TANF policies (e.g., sanction policies, work requirements) and in underlying economic, social and demographic conditions. Finally, comparisons across studies are even more problematic when looking beyond the ASPEfunded studies, because of the many differences in study populations, time periods studied, sources of data, and research methodologies.
Interestingly, despite the many differences in studies, the preliminary findings from the four studies remain quite consistent, particularly in the areas of employment and recidivism.
Over the 12-month period, some former recipients lost their jobs, while others found new employment, resulting in a cumulative employment rates of 68 to 82 percent, measured as those who were ever employed within the first 12 months of exit (see Table 1).
| Grantee | Exit Qtr | 1st Qtr post exit | 2nd Qtr post exit | 3rd Qtr post exit | 4th Qtr post exit | Ever Employed within 1 year |
|---|---|---|---|---|---|---|
| Arizona | 60.9 | 58.2 | 55.8 | 55.1 | 55.4 | 74.7 |
| Cuyahoga Co. | a/ | 59.3 | 54.2 | 55.8 | 56.8 | 71.7 |
| San Mateo Co. | 50.5 | 49.6 | 49.9 | 48.4 | 50.3 | 82.3 |
| Washington | 55.0 | 52.0 | 52.0 | 52.0 | 52.0 | 68.2 |
a/ - Data not available
These employment rates are consistent with findings in many other leavers’ studies, although rates are somewhat higher in some studies. These higher rates are largely explained by methodological differences (e.g., the use of survey data and/or the restriction of the study population to leavers who do not return to welfare). While these employment rates are similar to employment patterns for AFDC leavers in the past, they indicate a substantial increase in the absolute number of families leaving welfare with earnings, given the large increase in case closings in the past few years.
| Grantee | 1st Qtr post exit | 2nd Qtr post exit | 3rd Qtr post exit | 4th Qtr post exit | Ever Receiving within 1 year |
|---|---|---|---|---|---|
| Arizona | a/ | a/ | a/ | a/ | 28.4 |
| Cuyahoga Co. | b/ | b/ | b/ | b/ | 36.7 |
| San Mateo Co. | 8.2 | 13.3 | 13.3 | 14.1 | 24.4 |
| Washington | 12.0 | 19.0 | 22.0 | 23.0 | 29.8 |
a/ - Arizona reported the percentage of leavers returning to TANF in each quarter, unlike San Mateo and Washington which reported the percentage of leavers receiving TANF in each quarter, regardless of when they returned.
b/ - Cuyahoga County reported the percentage of leavers who had ever received TANF during the first 12 months after exit; they did not report quarterly data.
Most of these re-enter in the third month itself, since cases that re-open after one or two months were excluded from the study population. The proportion of former recipients receiving TANF increased to between 13 and 19 percent at two quarters after exit, and then increased more slowly, reaching 14 to 23 percent one year after exit. The proportion that ever returned for at least one month over the first 12 months after exit is somewhat higher, ranging from 24 percent in San Mateo County to 37 percent in the more urban environment of Cuyahoga County (Cleveland) in Ohio. Again, these findings are in line with other analyses of administrative data.
Interim reports from Cuyahoga County and San Mateo County suggested lower levels of program receipt, but differences in research methodology preclude direct comparisons.(2) Further explorations of food stamp and Medicaid receipt are expected in the final reports. In addition, the final reports will include analysis of what happens to families leaving welfare a couple of years after TANF implementation (the early findings are from late 1996), including analysis of surveys that include additional measures of child and family wellbeing.
In addition, a number of states have undertaken nonASPEfunded projects to collect data on what happens to welfare recipients and former recipients over time, as welfare programs change and as program participants move into the job market and/or lose their welfare benefits. These studies vary substantially in terms of study design, cohorts, administrative data linkages, research topics, and response rates(3). The ASPEfunded state/county grant projects are designed to improve the quality and consistency of approach of such research across states.
Among ASPE’s other projects, only one study, supported in part by welfare outcomes funds in FY 1998, has produced any results to date.(4)
The major purpose of the 1998 grants was to enable grantees to track and monitor how individuals and their families do in the first year after they leave welfare and to provide a foundation for longer follow-up. Grantees are using a combination of linked administrative data sets and surveys of samples of former recipients to monitor welfare outcomes. A large number of potential subgroup analyses were proposed by grantees, although their ability to carry them out will depend on sample sizes and survey response rates. Many grantees planned to stratify their samples based on their most important subgroups in order to assure sufficient sample size. The most common subgroups were based on reason for exit: sanctions, time limits, employment, and other. Several grantees planned to look at differences in outcomes for cases in urban versus rural areas. Many other proposed subgroup analyses were based on demographic characteristics such as age of recipient, age of youngest child, race or ethnicity, and prior education or work experience.
Through administrative and survey data states are addressing a number of important research questions in eight general topic areas including: employment and earnings, other income supports, health insurance, child care, child wellbeing, case closures and recidivism, barriers to selfsufficiency, insecurity/deprivation, and other topics. ASPE anticipates making approximately $1 million of (the $5 million) FY 1999 funding available to provide continuation funding to selected 1998 grantees. Brief descriptions of each FY 1998 grant project, including estimated project completion dates, are included in Appendix A. As noted above, interim reports, providing preliminary findings about former TANF recipients in the areas of employment, returns to TANF, and participation in other programs, have been released by four grantees as of midMay and are expected from the remaining grantees within another month or two. More detailed findings, including information gathered through survey data, will be presented in the final reports, which should be forthcoming from most grantees in late 1999 and early 2000. Two grantees (New York and San Mateo County) proposed a twoyear project period, with secondyear funding subject to grantee performance and availability of funds.
State welfare reform waiver studies are an integral part of the on-going welfare reform and social sciences research and evaluation agenda supported by the Administration for Children and Families (ACF) under section 1110 and section 413(h) of the Social Security Act, as amended. A central focus of ACF’s overall welfare reform research and evaluation strategy is to develop reliable, credible information about how different strategies are working in order to inform the flexible state policy choices permitted under the Temporary Assistance for Needy Families (TANF) program. Moving families from welfare to work, persistence and progression in employment and the wellbeing of children are major focal areas. In addition to examining the impacts of welfare reform on families and children, issues affecting the affordability and quality of child care and labor force attachment for lowincome families are central topics. Similarly, studies have been initiated to increase understanding of the effects of transitions in health and human services on other special populations — teens, Native Americans, victims of domestic violence, and rural populations among others.
Welfare reform waiver studies are part of this broad agenda. Prior to TANF, 43 states were granted waivers to demonstrate new approaches in welfare with agreements to conduct rigorous evaluations using random sample study designs. By comparing the experimental and control groups over time in the welfare reform waiver studies, the causal effect of a state’s reform effort on such key outcomes as welfare dependency, employment and earnings, total family income, and family structure can be reliably determined. Similarly, studies begun as state waiver demonstration evaluations prior to TANF which have been modified continue to be of interest. This mix of evaluations will allow the Department to examine the process of, and in some cases the impacts related to, implementing a variety of state approaches to welfare reform, including policies related to work and personal responsibility and time limiting assistance.
Given the breadth of policy choices available to states under the TANF program without waivers, and the fact that 23 welfare reform waiver studies are being supported with other Department funds, ASPE determined that significant new investments in waiver studies are not necessary at this time. The grant awarded to South Carolina (mentioned above and described in Appendix A) in 1998 to determine the status of TANF recipients after they leave the TANF caseload is an expansion of its welfare reform waiver study. Four other states (Arizona, Florida, Illinois and Wisconsin) that received 1998 grants to study the status of TANF recipients after they leave the caseload are also conducting waiver studies.
After extensive discussions with USDA’s Food and Nutrition Service concerning case information in the National Integrated Quality Control System (NIQCS) and state food stamp administrative records, it has become clear that significant technical obstacles must be overcome in order to use food stamp administrative data to look at welfare leavers and nonparticipants. For example, welfare “leavers” must be observed at two points in food stamp records — at a point when they are receiving both AFDC/TANF and food stamps, and at a point when they are receiving just food stamps. Since the NIQCS information is based on a pointintime review of food stamp participation, rather than a longitudinal look, one approach would be to merge state case record data with the NIQCS files to create a sample for review. However, it was ultimately determined that merging such data would not be feasible for several reasons, not the least of which was the burden that would be placed on state agencies to identify and retrieve relevant case records.
Therefore, the project, still in the conceptual design stage, will explore various other ways in which the food stamp data can be used to look at outcomes regarding employment and continued participation in other programs. This could include various analyses depending on their feasibility, such as an assessment of whether TANF leavers and nonparticipants in the FY 1998 grantee states and counties are representative of leavers and nonparticipants in the rest of the country. Another example of potential analysis under this study is a simulation of changes in TANF participation rates among those who are income eligible and on Food Stamps.
Number of Grantees Using Data Source
Source: Grantee revised work plans, as of 1/99.
Most of the wage records will come from the Unemployment Insurance system, though at least two will use their State Department of Revenue records. Child welfare, child support, and JOBS or JOBS successor data will be analyzed by about half the grantees. Other administrative data sets used by fewer grantees include child care subsidies, general assistance, SSI, substance abuse services, publicly funded mental health services, WIC, housing assistance, vocational education, school attendance, emergency services, and JTPA data. Some grantees will be linking fullcaseload samples, while others will link only subsamples. Some plan to use longitudinal data reaching back as far as 10 years, while others will include data only from 1997 forward.
State proposals for grants under both the leavers and child indicators solicitations were evaluated along many dimensions to ensure the collection of information on welfare outcomes from a variety of disparate sources. For example, grants were awarded to large urban areas and rural states (Los Angeles County and West Virginia); to states with large and virtually nonexistent immigrant populations (Florida and Vermont); and to states whose welfare benefit levels varied greatly in relation to poverty guidelines (18 percent of poverty in South Carolina and 66 percent of poverty in Alaska). ASPE also attempted to include a wide range of economic conditions among the grantees.
Despite the generally widespread and sustained period of economic growth throughout the country, ASPE considered several different measures of economic conditions as proxies for low and high growth economic areas. Among those measures for which state data are available were the unemployment rate, the rate of employment growth, and changes in the “not employed” (unemployed plus “not in the labor force”) rate. Ultimately the Change in Employment/ Population Ratio was determined to be a good measure for purposes of studying welfare outcomes. This ratio compares job growth to population growth in a state — a positive change in the ratio means that the number of new jobs is growing faster than the population (including but not limited to new job seekers); a negative change in the ratio means that the population is growing faster than the number of new jobs. ASPE also examined changes in the employment/population ratio over three time periods — 19961997 (the most recent period for which data were available when grant applications were being evaluated in 1998), 19951997, and 19941997. Given the wide variation in small states’ performance from year to year, ASPE concluded that using the threeyear average would be a more stable and reliable measure of actual state performance.
Figure 3 displays the percent changes in the Employment/Population Ratio from 19941997 for all states. Of a total of 23 leavers and/or child indicators grantees, ten states (Maine, Washington, Missouri, West Virginia, Massachusetts, New York, Rhode Island, Wisconsin, Georgia and California) are among the top third in performance on the Employment/Population Ratio, with a positive ratio of 3.0 or greater. Seven states (Delaware, Utah, Arizona, Hawaii, Minnesota, Alaska and Iowa) and the District of Columbia, had negative ratios, and are among the bottom third of performers.
Note: An “L” next to the state name denotes receipt by the state or some counties of a grant to study welfare outcomes (“leavers”); a “C” denotes a state receiving a grant to develop child indicators.
Source: U.S. Department of Labor, Bureau of Labor Statistics, Local Unemployment Statistics, Geographic Profile of Employment and Unemployment, annual, (data available online at http://stats.bls.gov/).
As directed by the conferees, ASPE funded the National Academy of Sciences (NAS) in 1998 to conduct a Panel Study to evaluate the design of current, proposed and future studies of the effects of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996 and make recommendations on further research. Through a combination of workshops, quarterly panel meetings and written reports, the NASappointed panel of experts continues to: (1) review our existing research efforts on the effects of welfare reform; (2) examine the appropriate data sources, research designs and statistical methods for analyzing welfare reform outcomes; and (3) identify research gaps and data needs for the continued study of welfare reform.
Funding of the National Academy of Sciences Expert Panel will continue in 1999 as part of ASPE’s overall research strategy for studying welfare outcomes. Over the course of this 30-month Panel Study, the Academy will continue to conduct workshops and will publish both an Interim Report and a Final Report. The Interim Report, due this summer, will contain a summary of the Panel’s first workshop as well as the Panel’s early recommendations on the Department’s welfare reform research plan. The Final Report will document the Panel’s comprehensive list of final recommendations on welfare outcomes research.
The projects supported by the targeted Policy Research funds are critical for understanding the outcomes of welfare reform and the Department is committed to advancing its welfare outcomes research data base. The research, evaluation and data activities supported by these funds are also crucial to the Department’s ability to respond to questions about the outcomes of welfare reform. The Administration for Children and Families and the National Institute of Child Health and Human Development, in particular, have contributed both financial and intellectual resources to enhance these efforts. Substantial support for the state and local grant projects was also provided by the U.S. Departments of Agriculture and Labor.
Projects proposed to be funded by the $5 million FY 1999 appropriation are summarized in the next chapter. The final chapter of this report provides an update on projects funded by the $5 million FY 1998 appropriation. Details on the 1998 state and county grants awarded for welfare “leavers” projects are included in Appendix A. Descriptions of the grants awarded to advance states’ child indicators initiatives are included in Appendix B.
1. Arizona’s interim report provided information on families who left welfare in the first (calendar) quarter of 1998; the data shown for Arizona in Tables 1 and 2 are on families who left welfare in the fourth quarter of 1996, and were provided to ASPE in a supplemental report. The data shown for San Mateo County and Washington in Tables 1 and 2 are also for the fourth quarter of 1996. The data shown for Cuyahoga County are for welfare leavers in the third quarter of 1996. The Cuyahoga County and Washington interim reports reflect families who left welfare prior to TANF implementation, which occurred in October 1996 and January 1997, respectively. Arizona implemented TANF in October 1996 and San Mateo County in November 1996, and reported data for the fourth calendar quarter of 1996.
2. For example, some of the studies appeared to have reported receipt of food stamps and Medicaid among leavers who did not return to TANF, while others looked at all leavers. Also, some studies reported Medicaid receipt for adult leavers, while others examined receipt for children, who are more likely to receive Medicaid. In a statefunded report of survey findings, for example, researchers in Washington state reported that 44 percent of adult leavers and 64 percent of their children were covered by Medicaid three months after exit.
3. The General Accounting Office examined 17 reports based on studies conducted or sponsored by states of families who left the AFDC or TANF rolls during or after 1995 and summarized the findings of seven of those studies in a report WELFARE REFORM: Information on Former Recipients’ Status (GAO/HEHS-99-48) in April 1999. In addition, the Urban Institute summarized the findings on employment rates, characteristics of employment and other determinants of wellbeing from 11 state studies of leavers in May 1999 (Where Are They Now? What States’ Studies of People Who Left Welfare Tell Us, a product of Assessing the New Federalism, Series A, No. A-32).
4. Preliminary analyses of Los Angeles County administrative data and the Census Bureau’s Current Population Survey show that the participation rate of immigrants in a wide array of benefit programs has declined more precipitously than the rate of citizens, even prior to implementation of new immigrant eligibility restrictions. Additional information on that study, “Understanding the Impact of TANF and Other Laws on Immigrant Families,” can be found in the third chapter of this report.
5. Separately funded research recently released by the George Washington University Center for Health Policy Research found that formal strategies to divert families from welfare are an increasingly common aspect of states’ efforts to shift to a workoriented assistance system. More than half the states have instituted formal programs to avoid enrolling families in welfare programs by finding other ways to assist them, or by requiring workrelated activities at a much earlier point in the application process. The researchers also reported an increasing awareness and concern that the implementation of aggressive state welfare reform efforts can lead to decreased access to other public benefits such as Medicaid.
[ Report’s Main Page | 1999 Projects | 1998 Projects | Appendix A | Appendix B ]
[ HSP Home Page | ASPE Home Page | HHS Home Page ]