A Summary of the Meeting of May 30-June 1, 2001. How to Train Community Partners to Use data and How to Identify and Deal With Pitfalls


This session was coordinated by Bong Joo Lee of Chapin Hall. The principal speaker was David Murphey of the Vermont Agency of Human Services. Representatives from Utah, Georgia, New York, and Vermont talked about how they have addressed the issue in each state, then led a discussion on some of the common key issues. In introducing the session, Murphey said that all states participating were behind the idea of a community curricula for data, although perspectives on what that meant varied. Murphey also pointed to key issues that states have identified that he expected to be discussed, such as the small numbers that arise when data are taken down to the community level, program outcomes versus community outcomes, and issues around the idea of making comparisons.


Terry Haven from the child advocacy organization Utah Children spoke first. Utah Children is the state's Kids Count grantee. They gather community indicators by zip code, working with the Department of Health, and use the data to advocate at the state level. Utah is very conservative and decisions are strongly influenced by conservative religious beliefs. Within the state, powerful conservative groups have strong lobbying organizations. Communities need to get data that they can advocate on their own behalf.

Utah Children runs an Advocacy Academy for nonaffiliated community people to attend. It is a three-day training workshop, free of charge. Transportation, meals, and accommodations are paid for. Participants learn to do things like deal with the media and organize at the grassroots level using Kids Count data. They are required to do at least one data presentation in their community and one community-based organizing advocacy effort in their community. Among Utah's special circumstances is that the percentage of kids in the population is the highest in the U.S., so even a relatively small percentage of children in adverse circumstances is still a large number. It is critical to present these data to the layperson so that people understand that small percentages may mask large numbers.


Rebekah Hudgins from Family Connection presented. Family Connection is a statewide network of community collaboratives that was started with support from Pew, Casey, and other foundations. Each of the 159 counties in Georgia now has a collaborative made up of people from different areas that work to make changes in decision making, service integration, and ultimately the well-being of children and families. There are 26 benchmarks that the collaboratives are working toward. Statewide, data have been collected on 19 of these benchmarks. All of the counties already have available to them, on a web site, county-level data related to all the state benchmarks.

State and federal money support a state-level infrastructure to provide technical assistance (TA) and support work at the county level. In 1991, there were 14 counties involved. In 1996, there were 40. This year is the first year that all 159 counties are involved. All counties receive the same level of state resources. The resources available to support this work have stayed constant, although the number of collaboratives has grown. TA teams were created to serve targeted purposes, such as evaluation or planning. Twelve regional consultants act as liaisons between the counties and the state and those consultants help counties identify their TA needs and structure TA teams to meet those needs.

The teams began with a state-level approach and, using the resources of several consultants, developed a handbook that gave basic background information on a variety of topics, such as defining evaluation, how to use evaluation, and how to use data. The handbook also provides sample data collection instruments. Also available were workshops within regions on how to use the handbook, how to use the web site, and how to provide feedback on self-assessment. The self-assessment is a tool completed by each county every fiscal year to provide feedback.

New York

The New York Council on Children and Families is a state agency that plays a coordinating role only. The Council thinks of itself as a membership agency, the members being the heads of the relevant state agencies.

New York has not yet done any training on how to use data, but expects to. The Council is at work developing a curriculum with the goal of enhancing the ability of state, regional, and local planners to effectively use health and well-being indicator data to guide health education. They have a web site, called the Kids Well-being Indicators Clearinghouse (KWIC) that will soon be available for members to access KidsCount data. In designing that web site, they asked data users and technology people what would be useful in a web site. It includes information on how to prepare communities to use data, and how much data you need to support good decisions.

The training session, as currently envisioned, would take place over one to two days, depending on the amount of technical information required, and would be in a traditional format and led by an instructor. The state has used both technical and community advisory groups to help design this training and curriculum.

This activity has run into a number of challenges in curriculum design and logistics and in piecing together funding from members. The Council has tried to address a number of troublesome issues, including the difficulty of moving from objectives to measures and helping people understand why data is being collected and how it can be used.


Vermont is similar to Georgia in that they have twelve regional partnerships. Each partnership receives $1600/year to buy training and TA from a menu of options. The menu approach allows people to get information when they need it.

Recurring Themes Across States

Language. It is important to agree on a common language within the state (For example, how to talk about outcomes vs. results vs. indicators.)

Accessing available data. Communities often don't know what is already available.

Collecting new data. States need to develop skills in different data collection methods.

Interpreting data. States need to know what questions to ask and how to recognize/interpret less obvious factors. For example, if in-hospital data shows fewer injuries result in death, is this because there are fewer injuries, or because of managed care and better outpatient care?

Making comparisons. Comparisons with the same data element over time are the most valid, but comparisons with peer groups or like communities is more difficult. People who work with indicators may need help identifying those communities.

Statistical significance testing. Those presenting data need to always provide confidence intervals and instructions on using and understanding the data and to temper the statistical information with explanations of practical significance and limitations.

The reasons for having indicators are to inspire change. It is important to choose priorities and to know what works and actually tackle it. The necessary resources to address a problem are more than just money; political will is important too.

Use of Indicators to Track Welfare Reform

Session coordinator was Larry Aber of the National Center for Children in Poverty (NCCP) at Columbia University. Speakers included Aber; Catherine Walsh, Program Director of Rhode Island Kids Count; and, Martha Moorehouse of ASPE.

Aber used two Powerpoint presentations, one on the work of the Research Forum on Children, Families, and the New Federalism, which is headed by Barbara Blum, and the other on indicators of social exclusion among children. These follow Aber's section of the text, coming before the summaries of the other speakers.


Larry Aber

The Research Forum on Children, Families, and the New Federalism, housed at the NCCP, is a repository of information on outcomes for families under TANF. A goal of centralizing information at the Forum is to promote syntheses of information to inform midcourse corrections. Aber expects the debate over reauthorization of TANF to emphasize two of the goals  preventing and reducing non-marital pregnancy, and encouraging the formation and maintenance of two-parent families. There is not much research about these areas.

Caseload Dynamics

There are many possible explanations for the decline in welfare caseloads under TANF. Movements in the work force, departures due to sanctions, reduced entries to welfare, and the strong economy  all could help account for the drop in caseloads. Welfare waiver experiments allowed the study of some of these factors. Nonexperimental administrative and survey data do not.

Findings about the way welfare programs have changed under TANF so far:

  • Cash assistance, use of food stamps, and use of Medicaid have diminished.
  • The legislative intent to promote job entry and work seems to be achieved. Many issues related to job retention and the adequacy of income remain.
  • The decline in nonmarital pregnancy and divorce rates, which began prior to Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA), continues. It is important not to credit TANF solely with these drops.

Little is known about effects on children, immigrants, or other populations that were outside the studies' principal foci. We don't know what will happen under a soft economy.

Research studies from the 1980s indicated that modest investments correlate with modest improvements and that caseload dynamics are related to education and employment histories. Three early and sobering controlled experiments  New Chance, TPD, & LEAP(1)  showed difficulties in improving education, employment income, and childbearing outcomes for young mothers. The New Chance study showed in part the prevalence and severity of physical and developmental problems for children of teenage mothers.

Waiver studies in the 1990s have been documenting increases in employment and reductions in caseloads at sites where time limits and sanctions were imposed. Incentive programs can reduce caseloads, improve employment and earnings, and reduce poverty at the same time. Some of these have effects on child development as well, highlighting a key issue  how do we improve income when people make the transition from welfare to work? It is expected that two research initiatives, the Project on Devolution and Urban Change and the Three City Studies are going to yield information derived from multiple sources  surveys, administrative data, and ethnographic research. As reauthorization approaches, there is an opportunity to apply what we have learned to national and state policy and local practice activity. From Aber's perspective, most of the compelling data come from the experimental welfare waiver studies.

A Study by the National Center's Social Science Research Unit

Aber and his colleagues have made an effort to use nonexperimental trend data to study welfare reform on family economic well-being at the national level.

Research Questions

  • What has been the effect of TANF (not pre-TANF waivers) on the economic well-being of children and how has that effect differed from effects associated with waivers?
  • How much of the effect of TANF is dependent on parent characteristics?
  • How do different parts of the TANF package affect economic well-being.

Study Data

The NCCP team has taken advantage of the Current Population Survey design that reinterviews half the sample at 12 months, allowing the analysis of short-term, one-year changes in income that can be related to policy changes. A central methodological difficulty is finding some of the respondents for reinterview. Upwards of 35 percent of respondents might be not locatable. This creates selection bias and poses the question  are these 35 percent different and if so, how? The NCCP team has analyzed that bias and found that those unlocatable for the second interview are more likely to be Hispanics, high school dropouts, people on welfare or below the poverty line, and people in mother-only families. The NCCP team believes that their work to compensate for this selection bias has yielded a dataset suitable for analyses to drive policy.


In examining changes in families' income-to-need ratios at different periods of time from 1988 to 1999 the two major policy variables were:

  • Were the respondents in a state implementing a welfare waiver?
  • Were the respondents in a state implementing TANF?

They found that TANF and waivers both affected people with less than high school education differently than they affect those with high school education. And waivers seem to have a bigger effect on family income than TANF did. Aber suggests that one reason for this difference is that states customized their programs under waivers to their particular circumstances. Aber thinks these findings confirm their prediction that TANF has advantaged the relatively advantaged among the poor and relatively disadvantaged the most disadvantaged among the poor. Also, they don't find that some provisions of welfare reform, such as sanctions, terminations, and family caps have an effect on family income. What has had an effect on income is earnings disregards. Aber suggested, in response to a question, that health or state population survey data might allow for similar analyses within states.


Catherine Walsh

At the beginning of her presentation, Catherine Walsh noted the legislative intent of the Rhode Island's Family Independence Act (FIA) and identified some indicators that reflect on those intentions and supporting data. Those intentions  or expected outcomes  for FIA are:

  • Increase in family income through employment
  • Gradual decrease in the level of cash assistance to employed families
  • Gradual decrease in state expenditures for cash assistance for all families
  • Reinvestment of cash assistance savings from family earnings into health care, child care, education, literacy, and skills training
  • Enhancement of family cohesion and stable living environment for all children

The FIA goals do not include caseload reduction. Although Rhode Island has shown smaller caseload reductions than some other states, they feel this is due to the income disregard provision of the Act. The costs of cases to the states have declined while family incomes have risen. Walsh showed how indicators are presented in context in the Rhode Island's Kids Count Fact Book. It is key that the reader understand why each indicator is important. As an example, she discussed the way they group and report the indicators for the five Rhode Island communities that have 25 percent or more of their children living in poverty (they call these the core cities), so that their circumstances aren't overlooked.

Rhode Island uses welfare reform indicators to describe child, family, and community conditions; inform planning and policymaking at the state and local level; measure progress in improving child and family outcomes; improve programs; and monitor the impact of policy choices. She said that the first point, to describe the conditions, is very important. Rhode Island tracks all indicators at the local community level. Management of data from these services is easy because a single state agency, the Department of Human Services, is responsible for implementing and overseeing the state's Family Independence Program (FIP). Activities under the program include case management, cash assistance, childcare subsidies, and health care coverage.

Three principles guide Rhode Island's welfare reform efforts: 1) Poor children should be no worse off than they were before welfare reform; 2) Adults should be able to access education and training if they need it before they are required to work; and 3) The program of cash assistance and supports for families should be cost neutral during the first two years (1997 and 1998).

Indicators and Outcomes

Progress toward the outcomes for FIA specified above can be tracked by a number of indicators. Walsh identified some of these.

Increase in family income through employment.

  • Average wage at job entry
  • Wages of "FIP leavers" versus "on FIP and working"
  • Job retention rates
  • Increases in earnings over time
  • Access to work, education, and training for public housing residents
  • Work participation rates

Gradual decrease in the level of cash assistance to employed families.

  • Percentage of cases with adults working
  • Average monthly cash benefit cost per case
  • Percentage of newly eligible cases working
  • Percentage of FIP cases with no employment history while receiving cash assistance

Gradual decrease in state expenditures for cash assistance for all families.

  • Annual federal and state expenditures for cash assistance

Reinvestment of cash assistance savings from family earnings into health care, child care, education, literacy, and skills training.

  • Annual federal and state investments in health care
  • Number of children in low-income working families and FIP families enrolled in Rite Care
  • Annual federal and state investments in child care
  • Number of children in low-income working families and FIP families receiving child care
  • Number of available child care slots

Enhancement of family cohesion and stable living environment for all children

  • Number of minor teen heads of household enrolled in FIP
  • Percentage of FIP caseload in two-parent households

Regarding the last goal, enhancement of family indicators, Walsh wishes to place the child development concerns centrally in the area of influence for this indicator so that it is part of the welfare conversation at the policy level. She wrapped up her presentation by pointing out some areas for indicator development and data gathering. These included indicators on how welfare reform:

  • Assists families in obtaining sustainable jobs that move them out of poverty and into economic self sufficiency
  • Supports the healthy development of children
  • Provides the range of supports and services needed by low-income families

She then provided a brief list of resources that provide models for indicator development and possible questions for statewide surveys. These included the National Survey of American Families and America's Children National Indicators.


Martha Moorehouse

Martha Moorehouse began by noting that the indicators project began because ASPE wanted to ensure that, in the tracking of welfare reform, children were part of the story  and not just children in welfare families, but other low-income children as well. Indicators need to be in place to capture the experiences of both groups and to explain how low-income children are faring generally, with welfare reform as backdrop rather than foreground.

Her feeling from looking over available presentations from studies on welfare reform is that including children is now the norm. But she has not seen much on children at the state level. Moorehouse said that analyses of data from experimental studies of welfare reform were presented at the New World of Welfare Conference. Some of those papers are available at the website of the University of Michigan's Gerald R. Ford School of Public Policy (www.spp.umich.edu). She went on to comment on a particular paper from that conference, "Welfare Reform and Child Well-Being" by Lindsay Chase-Lansdale and Greg Duncan. She commented in particular on a section of the paper which focused on achievement by children and adolescents in participating families. Regarding children, welfare experiments that provided families with generous earning supplements yielded higher achievement among children than those with no earning supplements. As a corollary, she pointed out that the Minnesota experiment, which showed some of the higher rates of achievement among children, did not continue the generous earning supplement under TANF. Moving on to adolescents, more problem indicators seemed to be emerging. Moorehouse concluded by saying that, so far, welfare reform is not doing for children what it should, but that there seems to be new interest in what needs to be done for kids in welfare families and the broader low-income population. States should pursue data that will allow them to decide whether other interventions are in order.


1.  The New Chance demonstration project provided education, training, and other services to women who had children as teenagers and dropped out of high school. New Chance was intended to increase the long-term self-sufficiency and well-being. It was evaluated by Manpower Demonstration Research Corporation, which released its final report in 1997. Ohio's Learning, Earning, and Parenting Program (LEAP) used financial incentives and penalties, combined with case management and support services, to promote school attendance by pregnant and parenting teenagers on welfare. MDRC evaluated LEAP, issuing a report in 1997. The Teen Parent Demonstration Program was a case management and mandatory education, job training, or employment program for first-time teen parents (or other pregnant teens in some jurisdictions) on welfare. TPD was evaluated by Mathematica Policy Research, which issued a report in 1998.