Appendix A:
Evaluating Welfare Reform in an Era of Transition
Conclusions and Recommendations
[ Main Page of Report |
Contents of Report ]
Panel on Data and Methods for Measuring the Effects of
Changes in Social Welfare Programs
Robert A. Moffitt and Michele Ver Ploeg, Editors
Committee on
National Statistics
Commission on Behavioral and Social Sciences and Education
National Research Council
National Academy Press
Washington, DC
Chapter 3
Research Questions and Populations of Interest
-
Conclusion 3.1
-
The primary population of interest for measuring the effects of changes in
social welfare programs is the low-income population. The primary group of
interest to the TANF program is the population of low-income mothers and
their children.
-
Conclusion 3.2
-
Within the low-income population, those groups who have been on welfare or
who are eligible for welfare are of particular interest. Within the population
of welfare eligibles, there are four separate subgroups, each of which is
of special interest for welfare reform studies: those who leave welfare,
those who stay on welfare, those who are formally diverted from welfare through
diversion programs, and those who are poor but have not applied for benefits
or who have applied but been rejected.
-
Conclusion 3.3
-
The specific service needs of some low-income individuals and families also
define subpopulations of interest for welfare reform research. First among
these are families with special circumstances or characteristics that make
the transition to employment and self-sufficiency difficult. Other subgroups
of the low-income population have special needs that require assistance
independent of their effects on employment, including: families with poor
physical or mental health, substance abuse problems, or problems of domestic
violence, as well as families with troubled adolescents or children with
special physical, cognitive, or behavioral problems.
-
Conclusion 3.4
-
The set of outcomes of interest for studies of welfare reform should be defined
broadly to include all the outcomes that the different audiences of studies
of welfare reform-the public, Congress and state legislators, and other
governmental officials and program administrators-are concerned about.
-
Conclusion 3.5
-
The monitoring questions of interest are the following: How has the well-being
of the low-income population and key subgroups evolved subsequent to welfare
reform? Which subgroups are doing well and which are doing less well? Which
subgroups are in greatest need and deserve the attention of policy makers?
-
Conclusion 3.6
-
The descriptive questions of interest regarding program policy and implementation
are the following: What policies, programs, and administrative practices
have states and localities actually implemented as part of welfare reform?
How wide is the variation across states and even within states in policy?
How has implementation differed from officially described policy? How has
the non-TANF programmatic environment changed?
-
Conclusion 3.7
-
The impact evaluation questions of interest are the following: What are the
overall effects of the complete bundle of changes in policies, programs,
and practices on the well-being of the low-income population, including the
effects on both adults and children and on specific subpopulations of interest?
What are the effects of the individual broad components of welfare reform
on the well-being of the low income population and subpopulations of interest?
What are the effects of specific detailed strategies within each of the broad
program components on the well-being of the low income population and the
subpopulations of interest--what works and for whom?
-
Conclusion 3.8
-
The effect of welfare reform is a question of interest for the nation as
a whole as well as for individual states.
-
Recommendation 3.1
-
The panel recommends that ASPE take primary responsibility for publicly defining
the questions of interest for welfare reform research and evaluation, identifying
emerging issues for social welfare programs, and defining alternative detailed
strategies and policies that address the what-works-and-for-whom questions.
In doing so, ASPE should expand its current activities in seeking input from
states, private foundations, and other stakeholders on emerging policy and
evaluation issues.
-
Recommendation 3.2
-
ASPE should produce an annual report to Congress that, among other things,
presents a comprehensive list of the important questions to be addressed
in welfare reform research, describes how those questions are being addressed
in the overall landscape of welfare reform studies, and explains how its
own research agenda relates to those questions and to other studies underway.
[ Go to Contents ]
Chapter 4
Evaluation Methods and Issues
-
Conclusion 4.1
-
Different questions of interest require different evaluation methods. Many
questions are best addressed through the use of multiple methods. No single
evaluation method can effectively and credibly address all the questions
of interest for the evaluation of welfare reform.
-
Conclusion 4.2
-
Experimental methods could not have been used for evaluating the overall
effects of PRWORA and are, in general, not appropriate for evaluating the
overall effects of large-scale, system-wide changes in social programs.
-
Conclusion 4.3
-
Experimental methods are a powerful tool for evaluating the effects of broad
components and detailed strategies within a fixed overall reform environment
and for evaluating incremental changes in welfare programs. However, experimental
methods have limitations and should be complemented with nonexperimental
analyses to obtain a complete picture of the effects of reform.
-
Conclusion 4.4
-
Nonexperimental methods, primarily time-series, and comparative group methods,
are best suited for gauging the overall effect of welfare reform and least
suited for gauging the effects of detailed reform strategies, and as important
as experiments for the evaluation of broad individual components. However,
nonexperimental methods require good cross-area data on programs, area
characteristics, and individual characteristics and outcomes.
-
Recommendation 4.1
-
The panel recommends that ASPE sponsor methodological research on nonexperimental
evaluation methods to explore the reliability of such methods for the evaluation
of welfare programs. Specification testing, sensitivity testing, and validation
studies that compare experimental estimates to nonexperimental ones are examples
of the types of methodological studies needed.
-
Conclusions 4.5
-
Existing household surveys are of inadequate sample size to estimate all
but the largest overall effects of welfare reform on individual outcomes
using cross-state comparison methods. Research is needed to address this
problem by considering the American Community Survey, state level administrative
data sets, and supplements and additions to the CPS or other surveys to increase
their capacity to detect welfare reform impacts in the future.
-
Conclusions 4.6
-
The problem of generalizability of the evidence from welfare reform evaluations
on specific populations, areas, and relationships to more general populations,
to a national level, and to new policies, has not been sufficiently addressed.
More use of microsimulation models as a tool to address generalizability
is needed. Microsimulation is also needed to assist in the synthesis of diverse
types of results.
-
Recommendation 4.4
-
The panel recommends that U.S. Department of Health and Human Services sponsor
process research in a number of service delivery areas to better understand
how service delivery administrations have implemented new welfare programs
and the benefits and services families and children are receiving under these
new programs.
-
Recommendation 4.5
-
Process and implementation studies have grown in number and importance in
the evaluation of welfare reform but often have design defects and are
insufficiently integrated with outcome evaluations. As a consequence, their
potential use in evaluation has not been fully reached. The panel recommends
that U.S. Department of Health and Human Services sponsor methodological
research on process and implementation studies to improve methods for
systematizing the documentation of program policies and practices, to develop
protocols and best practices, and to further integrate them with impact
evaluations.
-
Recommendation 4.6
-
Qualitative and ethnographic studies of the low-income population and its
relevant subpopulations and of social service agencies that provide services
to these populations are an important part of the overall welfare program
evaluation framework. The panel recommends the further use of well-designed
qualitative and ethnographic studies in evaluations of welfare programs to
complement other evaluation methods.
-
Recommendation 4.7
-
A welfare dynamics perspective should be incorporated into more welfare reform
studies, including leaver studies. In general, more disaggregation by levels
of heterogeneity among leavers and stayers is needed given the importance
of disaggregation for outcomes on and off welfare.
-
Conclusion 4.8
-
Studies of the outcomes of welfare leavers contribute only one part of the
story of welfare reform and, as an evaluation method, have been
disproportionately emphasized relative to other methods. Studies that compare
current leavers to those who left welfare prior to welfare reform and studies
of divertees, applicants, and nonapplicant eligibles need more emphasis.
-
Recommendation 4.9
-
More methodological research is needed to assess and improve the credibility
of the multiple cohort method of evaluating the overall effects of welfare
reform. This research needs to study the best method to control for the
time-series effects of other policies and the economic environment and how
many cohorts are enough to do this.
-
Recommendation 4.10
-
Experimental methods are underused in current designs of new welfare policy
evaluations and should be employed in future studies evaluating different
detailed reform strategies and different individual broad components.
-
Recommendation 4.11
-
The federal government should take a proactive role in sponsoring experiments
at the state and local levels and should encourage planned variation and
cross-state comparability to yield the maximum general knowledge.
-
Conclusion 4.5
-
Caseload and other econometric models have produced a mixed set of results,
partly because of data limitations and partly because of an inherent lack
of policy variability. They have done somewhat better at producing ballpark
estimates of the overall effects of welfare reform than at producing estimates
of the effects of individual broad components.
-
Recommendation 4.12
-
The federal government, taking all agencies as a whole, has produced and
funded a great deal of valuable monitoring research and a much smaller volume
of evaluation research. A greater effort to produce a comprehensive evaluation
framework for social welfare programs that considers the major questions
of interest and the evaluation methods appropriate for each is needed. A
comprehensive framework for evaluation should be developed and used to guide
the evaluation efforts under way by private and other public evaluation
organizations. This should be an on-going effort as new issues emerge and
is a responsibility that should be taken on by the ASPE in the U.S. Department
of Health and Human Services.
-
Recommendation 4.13
-
In its Annual Report to Congress, ASPE should review the existing landscape
of evaluation methods, whether the appropriate balance of experimental and
different nonexperimental methods is being achieved, and how evaluation
methodology fits into its own research agenda.
-
Conclusion 4.6
-
The panel finds that state capacity and resources to conduct evaluations
of their own welfare reform programs is often below the level is needed for
such an important change in policy.
-
Recommendation 4.14
-
The panel recommends that the U.S. Department of Health and Human Services.
continue and expand its efforts to build capacity for conducting high- quality
program evaluations at the state level through the provision of technical
assistance, convening of research conferences, promoting the exchange of
technical assistance among the states, and other capacity building mechanisms.
-
Recommendation 4.15
-
The panel recommends that ASPE be the primary agency responsible for synthesizing
findings from studies of the consequences of changes in welfare programs.
[ Go to Contents ]
Chapter 5
Data Needs and Issues
-
Conclusion 5.1
-
The panel finds that each of the major national household survey data sets
most suitable for monitoring and evaluation has significant limitations in
terms of sample size, nonresponse levels, periodicity, response error, population
coverage, or survey content.
-
Conclusion 5.2
-
Key national-level survey data sets used to monitor low-income and welfare
populations are currently not being produced on a timely basis. The value
of these data for monitoring low income and welfare populations would be
enhanced if they are produced on a more timely basis.
-
Recommendation 5.1
-
To improve the abilities of national-level survey data sets to measure the
effects of changes in broad welfare program components across states, the
panel recommends expansions or supplements to the CPS or other surveys.
-
Recommendation 5.2
-
A fully implemented and continuous American Community Survey has significant
potential for use in future welfare policy research. The panel recommends
that sufficient funds be devoted to fully implement the survey and that support
for the survey at its currently proposed sample sizes is sustained over time.
-
Recommendation 5.3
-
The potential of the American Community Survey for evaluating welfare policies
would grow considerably if the survey included more extensive questions about
public assistance benefit and service receipt. The panel recommends adding
more detailed questions on public assistance receipt to the survey questionnaire.
-
Recommendation 5.4
-
The wider array of services provided in social welfare programs and the variation
in these programs across states both make measuring program participation
and benefit receipt more difficult, especially on a national-level. For national
household surveys that measure participation in and benefits received from
programs serving the low income population, it is critically important to
regularly and frequently review survey questions to keep in step with program
and population changes. The panel recommends to the Census Bureau that more
resources be devoted towards improving questions on program participation
and benefit receipt to better capture program participation. The panel also
recommends that HHS work with the Census Bureau to develop mechanisms for
regular communication with states to stay abreast of programmatic and
implementation changes in the states.
-
Recommendation 5.5
-
State-level capacity to conduct household surveys of low-income and welfare
populations is limited. HHS has begun an important effort to build state
capacity for conducting surveys. These efforts need to be continued and expanded.
-
Recommendation 5.6
-
Administrative data, primarily at the state level, are an important emerging
source of information for both monitoring and evaluation. However, there
are many significant challenges that prevent them from fulfilling their
potential, including the conversion to research use from management use,
preservation of data over time, improvements in the quality of individual
data items, comparability of data across states, confidentiality and access,
and barriers to matching across different administrative and survey data
sets. Much more investment in this data resource is needed.
-
Conclusion 5.3
-
The lack of cross-state comparability is a major barrier to the use of
state-level administrative data sets for cross-state monitoring and evaluation.
-
Recommendation 5.7
-
The panel recommends that HHS, in conjunction with state social service agencies,
take steps to further improve the comparability of administrative data across
states. These steps should move toward comparable definitions of services
and service units and data formats. Building comparability across states
will have to be a cooperative effort between the federal government and states
and will likely require federal funding of state activities.
-
Recommendation 5.8
-
The current definition of assistance used to guide state data reporting
requirements is very narrow and will not capture many recipients receiving
different forms of assistance provided by states. The panel recommends that
the Administration for Children and Families consider broadening this definition
to include as many types of assistance and services provided as possible.
-
Recommendation 5.9
-
Administrative data reported by states as part of the TANF reporting requirements
will be of limited use for research purposes unless steps are taken to improve
them. The usefulness of these data will be improved if the data can be linked
to other data sets and if the full universe of cases is reported. The panel
recommends that ACF take steps to improve the linkability of these data and
encourage states to report the full universe of cases.
-
Recommendation 5.10
-
Confidentiality, privacy, and access concerns with administrative and survey
data and the linking of multiple data sets are important issues, but are
currently serving as a barrier to socially important evaluation of welfare
reform programs. The importance of access to these data for monitoring and
evaluation of programs should be emphasized and efforts to reduce these data
access barriers while protecting privacy and maintaining confidentiality
should be expanded.
-
Recommendation 5.11
-
The monitoring and documentation of the actual policies, programs, and
implementations of welfare reform at the state and local levels by the federal
government has been minimally adequate to date. The panel recommends that
the Department of Health and Human Services take active and direct responsibility
for documenting and publishing welfare program rules and policies in every
state and in every substate area where needed. Continuing updates documenting
changes in state and local area rules should also be produced.
-
Recommendation 5.12
-
In its Annual Report to Congress, ASPE should review current availability
and quality of data for welfare reform research, identify high priority data
needs, and discuss its own research agenda for data development and technical
assistance.
[ Go to Contents ]
Chapter 6
Administrative Issues for Maintaining the Data Infrastructure
-
Conclusion 6.1
-
No agency within HHS has distinct administrative authority and responsibility
for the collection and development of data relevant to social welfare and
human service policies and programs. This administrative gap is a major reason
for many of the inadequacies in the data infrastructure for monitoring and
evaluating welfare policies.
-
Recommendation 6.1
-
The federal government should be responsible for ensuring that high-quality
and comparable data on human service and social welfare programs and populations
are collected so that the well-being of the low income population can be
monitored and so that high-quality evaluations of the effects of welfare
reform can be conducted.
-
Recommendation 6.2
-
The panel recommends that an organizational entity be identified or created
within HHS and that this entity be assigned direct administrative responsibility
and authority for carrying out statistical functions and data collection
for social welfare programs and the populations they serve. The entity should
also coordinate data collection and analysis activities between states and
the federal government.
Where to?
Top of Page
Contents
Main Page of Report |
Contents of Report
Home Pages:
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)