Alternative Outcome Measures:
TANF Block Grant

Chapter III:
Examination of Selected Outcome Measures

[ Main Page of Report ]

Contents

• Employment-Related Measures
  • Job Entry Rate
  • Employment Retention Rate
  • Earnings Gains
  • Percentage of Those Required to Work with Earnings
  • Recidivism Rate
• Measures of Child and Family Well-Being
  • Food Stamp Receipt
  • Medicaid/SCHIP Receipt
  • Child Care Affordability and Quality
  • Receipt of TANF and Other Types of Transitional Assistance
  • Extreme Poverty Rate
• Family Formation/Stability Measures
  • Percentage of Children Living in Married-Couple Homes
  • Out-of-Wedlock Birth Rate for TANF Families

In this section, we consider selected outcome measures that could potentially be used to measure the performance of states' Temporary Assistance for Needy Families (TANF) programs. In order to assess the feasibility of using these alternative measures to gauge the success of state TANF programs, we examine several issues for each potential measure:

• What the indicator will measure and its relationship to the goals of the TANF program;
• Measurement issues that should be considered when defining how the indicator should be calculated;
• A brief discussion of the availability, quality, and timeliness of data to calculate the measure, particularly on a state-by-state basis (see Appendix D for more detail on a variety of potential data sources); and
• Issues related to the "fairness" of the measure, i.e., whether states are able to perform on equal footing with other states, and whether the TANF agency can reasonably be expected to affect the outcome. (The general questions of whether outcome-based performance measures accurately reflect program effectiveness, and how to adjust standards in order to compensate for differences in economic and socio-demographic factors are not discussed here.)

The measures considered below are not a comprehensive listing of possible alternative outcome measures, but are a representative sample of measures which focus on the goals that were of high priority to most participants in the consultation process: employment, child and family well-being, and the formation and stability of two-parent families. Within the context of these goals, we selected measures that seemed both salient and possible. In some cases, the proposed measures may be more accurately described as interim outcome measures or output measures, rather than true outcome measures. Several are based upon those measures being used for the TANF High Performance Bonus. We did not include measures that are already being used for other bonuses, such as the child support incentives, because the participants in the consultation generally agreed that it did not make sense to credit (or punish) states twice for the same performance under separate systems.
 

[ Go to Contents ]

Employment-Related Measures

Employment is one of the key objectives of the TANF program. At the federal level, the statute governing the TANF program explicitly defines ending dependence on government programs through the promotion of work as one of the four goals of the program. This is also one of the top priorities for states under TANF; when given the opportunity to compete for bonuses based on employment-related measures in FY 1999, a total of 46 states chose to do so. Because it is such a fundamental goal of TANF, multiple performance measures related to a state's success in helping individuals find and keep employment are discussed in this section.

Job Entry Rate

The job entry rate would measure the proportion of the unemployed TANF adult caseload that obtained a job. This measure gauges the success of states in achieving one of the key goals of the TANF program - moving individuals into employment. The job entry rate is one of the measures used by HHS to award the TANF High Performance Bonus. (The data used for making the FY 1999 awards are shown in Table 2.) In addition, the job entry rate has been used as a performance measure for workforce development programs operating under the Job Training Partnership Act (JTPA) and its successor, the Workforce Investment Act (WIA).

Measurement Issues. When defining the job entry rate, it is necessary to determine whether employment in both unsubsidized and subsidized employment will count toward the rate. Because it is perceived as more aligned with the goals of welfare-to-work programs, many programs using job entry as a performance measure generally count all employment that is not fully subsidized. In the interests of minimizing the data reporting burden, under the TANF High Performance Bonus final rule, HHS is counting all employment, whether or not subsidized.

Another measurement issue is determining which individuals to include in the rate. Should only those who receive benefits counting as "assistance" under the TANF rules be included, or should recipients of other types of services be counted? What about applicants who are diverted from receiving assistance? If diverted applicants are not counted, states with successful diversion programs will actually be lowering their job entry rate. Because there is no standardized definition of what constitutes an application, it would be extremely difficult to develop a measure that is comparable across states.

A related question is whether to count only individuals who leave cash assistance when they find a job or to also include those who remain on aid while working. Because individuals in states with low grant amounts are more likely to leave cash assistance when they find a job, states are treated more equitably if the rate counts all individuals who move into employment regardless of whether they leave cash assistance. Allowing only individuals who leave cash assistance for employment to count toward the rate may also give states incentives to reduce their grant levels or their earnings disregards. For these reasons, the job entry rate for the TANF High Performance Bonus allows states to count all individuals moving into employment, whether or not they are receiving cash assistance. However, finding work that ends dependence on cash assistance remains the ultimate goal of the TANF program.

A more expansive approach could examine work participation for a broader range of low-income families, not limited to those who have received TANF benefits. Such an approach would reward states that used their flexibility under TANF to serve a range of low-income families. However, this measure would probably be more affected by underlying economic conditions than by any action taken by the state's TANF program.

Data issues. The job entry rate could be measured on a state-by-state basis through several sources: state TANF administrative data, surveys, and Unemployment Insurance (UI) wage records. Of these sources, UI wage records, collected by state employment security agencies, are the preferable source, because they provide high quality employment data at a relatively low cost. However, they have two important limitations. First, they do not cover all jobs in a state, excluding such jobs as self-employment, agricultural employment, employment by the federal government or military, and jobs outside state boundaries. Second, UI records only track total quarterly earnings; they can not be used to calculate hourly wages. Historically, state administrative data on cash assistance recipients who find jobs have been of varying quality, in part because recipients often do not notify the welfare department when they find a job. Because data on job entry based on state TANF administrative records may vary in quality from state to state, it may be difficult to rank or evaluate state performance based on this source. Surveys may provide more accurate data, but they are generally too expensive for states to rely on for ongoing data needs.

Because of these concerns, for the 1999 and 2000 awards, states were allowed to submit the best data they had available for the interim High Performance Bonus measures, regardless of the source. Thus states could use matches with UI data, surveys, administrative records, or a combination of these data sources. In general, in 1999, states opted to use linked UI data, in some cases supplemented with information from administrative data regarding jobs not covered by the UI system. Under the final High Performance Bonus regulation, HHS has decided to calculate the work-related measures by linking information on TANF recipients with data from the National Directory of New Hires (NDNH), which combines the information in state UI databases with information from federal agency personnel offices. This minimizes the reporting burden on state agencies and ensures that the measures will be calculated based on data that are consistent across states.

Fairness issues. Data available to date for this measure (see Table 2 for FY 1998 data), show a great deal of variation in the job entry rates achieved. (For FY 1998, rates ranged from under 20 percent to almost 90 percent, with most states achieving rates between 30 and 50 percent.) This variation could be attributed to any or a combination of several factors, including the usual differences in economic conditions, the fraction of employment that is covered by UI in the state, and/or each state's stage of welfare reform implementation at the time of measurement. With respect to the last factor, states that had taken earlier aggressive steps to move recipients into work may have found that those recipients remaining unemployed faced substantial barriers to employment and thus were harder to place.

Employment Retention Rate

The employment retention rate would measure the length of time TANF recipients who found jobs stayed employed. In order to help individuals maintain self-sufficiency - a key objective of the TANF program - it is critical that states help individuals stay in their jobs or find another job quickly if they lose their initial job. Job retention is one component of the Success in the Workforce measure used under the TANF High Performance Bonus. (The data used for making the FY 1999 awards are shown in Table 3.)

Measurement issues. When defining a retention measure, it is necessary to consider whether the focus is on job retention - the amount of time an individual stays in a specific job - or employment retention - the amount of time an individual remains employed regardless of whether it is the same job or a different job. It may be preferable to focus on employment retention because, in some instances, a change to a different job may be a move to a better job.

Another issue in defining how to calculate the employment retention rate is determining how long individuals will have to remain employed in order to be counted in the rate. The TANF program is likely to be more connected to individuals during the initial period after they find jobs and may have more influence over employment retention in this early period. However, longer periods of retention are clearly more meaningful and desirable. Different retention period choices have been made in the past under various programs. The TANF High Performance Bonus for FYs 1999 and 2000 requires that individuals who find employment in one quarter retain employment into the next quarter. Beginning in the third year of this bonus (FY 2001), the "job retention" measure will be based on three consecutive quarters of employment. The WIA program measures employment retention six months after an individual initially finds a job. The JTPA program used a measure that combined job entry and retention by measuring the employment rate 13 weeks after program exit.

Data issues. The employment retention rate could be most easily measured on a state-by-state basis through UI wage records or at a national level through the NDNH. As discussed above, these records provide relatively comparable and timely data across states at a relatively low cost. States are currently not required to collect information on job retention for the TANF program's reporting requirements, and it would be costly for them to track such individuals. While the JTPA program measured job retention by conducting a survey 13 weeks after individuals left the program, in part because of the cost of this data collection, the successor WIA program will use UI data to calculate employment retention.

While UI or NDNH records can be used to determine whether an individual was employed in several consecutive quarters, they do not indicate whether the individual is in the same job, whether they left their initial job and found a subsequent job, and/or whether there was a break in employment (unless the break covered an entire quarter). Thus, these data can be used only to provide measures of employment retention, rather than job retention, over a certain number of quarters, but they will not capture certain breaks in employment.

Earnings Gains

The earnings gains measure would reflect the increase in earnings over a specified time period for employed TANF recipients. Individuals who experience earnings gains are more likely to sustain their self-sufficiency over the long run and to escape poverty. Earnings gains are included as a performance measure for the TANF High Performance Bonus (the data used for making the FY 1999 awards are shown in Table 4) and the WIA program.

Measurement issues. An important question is whether to measure earnings gains (which may be due to any of a number of factors - including an increase in the hourly wage rate, an increase in hours worked, or an increase in the number of days or weeks employed - or a combination of these) or to focus on an increase in hourly wages. Experimental studies have found that welfare-to-work programs have produced most of their earnings gains through increases in the amount of employment rather than through increases in hourly wages.

Another measurement issue relates to the time period over which earnings gains will be measured. The TANF program is likely to be more directly connected to individuals during the initial period after they find jobs, and may have more influence over recipient experiences in this early period. On the other hand, individuals who find jobs may have to spend some time in the labor force before they experience earnings gains. Welfare-to-work programs that use earnings gains as a performance measure typically do so over a relatively short time period. The TANF High Performance Bonus measures earnings gains between one quarter and the second subsequent quarter. The WIA program measures earnings gains over the six months after entry into unsubsidized employment, as compared to earnings in the period prior to WIA program participation.

Data issues. Like the employment retention rate, the earnings gains measure could most feasibly be calculated on a state-by-state basis through UI or NDNH records. While states usually have detailed earnings information in their TANF administrative data for current recipients, they do not systematically track the earnings of former recipients because it would be relatively expensive to collect. Thus, UI or NDNH records provide the highest quality data at the lowest cost.

Using UI or NDNH data, however, means that earnings progression can only be measured between one three-month period (quarter) and a subsequent one. These records do not provide data on the number of hours worked, and so they can not be used to measure wage gains. Moreover, because individuals may start or stop working in the middle of a quarter, earnings in a quarter may reflect a period of unemployment rather than a job with low pay or few hours. This is particularly likely to affect the first quarter of earnings.

Percentage of Those Required to Work with Earnings

This measure reports on the proportion of the TANF adult caseload that is required to work and that has earnings. This measure differs from those discussed above because it would measure the employment level only for those TANF recipients who are required to work and would include both individuals who find jobs as well as those who were already working. Because it includes all individuals on the caseload who are working, it is closely related to the current participation rate.

The statute governing the TANF program allows states to exempt certain individuals from the work participation rate - primarily individuals with a child under age one and disabled individuals in two-parent families - while the remainder of the caseload is required to work. In addition, states with a work program waiver in effect prior to the enactment of TANF are allowed to continue with their prior (and broader) exemption policies. Because it includes only individuals whom the state is attempting to move into employment, this type of measure may more accurately reflect a state's success in moving individuals into work compared to a job entry rate based on the entire adult caseload.

Measurement issues. A determination would have to be made whether those considered "required to work" would be measured according to the definition provided in the federal TANF statute or whether states operating under waivers with broader exemption policies could use their own definitions. Both definitions are problematic in some respect. If the federal definition is used, states operating under pre-existing waivers with broader exemption policies of their own may not perform as well on the measure because some of their exempt cases would technically be "required to work" under the federal statute. Similarly, if each state used its own definition of "required to work," it would be difficult to compare outcomes across states because of the differences in the populations being served.

Data issues. Like the other work-based measures, UI or NDNH records are the best source of data for this measure. For this measure the TANF adult caseload that is required to work (rather than the entire adult caseload) would be matched to UI or NDNH records. Since the final TANF regulations require states to track whether individuals are required to work, this should not impose a new or significant burden.

Fairness issues. There are a number of factors that would have uneven effects on state performance on this measure. As discussed above, the existing differences in state exemption policies because of pre-existing waiver policies create an unequal playing field for this measure. It may not be possible to compare state performance equitably given the differences in their current exemption policies. In addition, the generosity of earnings disregards and cash grant levels affect the ability and willingness of individuals to combine work and welfare when they find jobs. Finally, like the other employment-related measures discussed above, the health of a state's economy and the relative number of hard-to-serve (i.e., disadvantaged) individuals in its caseload will affect performance on this type of measure.

Recidivism Rate

The recidivism rate measures the extent to which individuals who leave TANF cash assistance return to the program within some specified time period. Because a key goal of the TANF program is to promote sustained self-sufficiency by ending dependence on assistance, the recidivism rate is an important measure of a state's success in achieving this goal.

Measurement issues. In defining the recidivism rate, it is necessary to specify the population that will be included. Recipients leave TANF assistance for a variety of reasons, including employment, full-family sanctions, time limits, and other reasons. If recipients leaving TANF for all of these reasons are included, a state's performance on this measure is likely to be more heavily affected by state policies than by its success in promoting self-sufficiency. For example, a state with many case closures due to a lifetime time limit is likely to have a relatively low recidivism rate. One way to try to level the playing field might be to limit this measure to those recipients who leave cash assistance due to employment. However, this approach raises its own set of problems; research shows that many more welfare recipients are employed after leaving welfare than administrative data records show as having closed their cases due to employment.

It may also be appropriate to limit the measure to cases that have closed for at least a minimum time period, in order to exclude those cases closed and then re-opened due to administrative churning (e.g., when a case is closed for failing to provide required information and re-opened a few days later when the client brings in the needed documentation). For example, in HHS-funded studies of TANF recipients who left assistance, state grantees were encouraged to define "leavers" as cases closed for at least two months, in order to minimize the effects of churning.

Another issue is that the members of a case may not all move on or off welfare together. For example, a case may be closed due to a full-family sanction, but the children in the case may later move into the custody of another family member and may receive assistance as a child-only case. A decision must be made whether recidivism means the return to welfare of the case head, of any adult in the case, or any member of the household, including children.

Data issues. At this point in time, state administrative records are the only reliable data source for measuring a recidivism rate. Because states routinely track the months that individuals receive TANF assistance, state administrative data should be of relatively high quality and available on a timely basis. While states are required to track the reasons individuals leave assistance, this measure would be somewhat more complex for states to calculate if they had to distinguish whether certain types of "leavers" (such as those subject to time limits or sanctions) - rather than if all leavers - were included in the rate.

Fairness issues. It may be difficult to develop a recidivism measure that treats states fairly and equitably. As noted above, state need standards and grant levels affect the ability and willingness of recipients who work part-time or at low wages to continue to receive assistance. This in turn affects an individual's job experience level (and the likelihood of finding another job quickly) when they leave assistance, as well as the relative attractiveness of returning to assistance if a job is lost. For example, individuals who leave welfare in a state with relatively high grant levels and/or generous income disregards may be less likely to return to the rolls - regardless of the effectiveness of the state's welfare-to-work program - because they are working at relatively high wages or for a greater number of hours at the time they leave welfare. Conversely, the higher grant level may make it more worthwhile to return to assistance if a job is lost. In a state with low grant levels and/or low earnings disregards, on the other hand, individuals will be forced to leave cash assistance when they find jobs even if they are low-wage, part-time, or temporary. With relatively less job experience, this group may be more likely to return to cash assistance over the long run, or they may feel it is not worth the hassle of returning to assistance for a small grant amount. Differences in state sanction policies and time limit rules will also affect how states perform on this type of measure. In addition, those states with a full-family sanction policy will have a different group in their "leaver" population than a state that only removes the adult from the case when a sanction is enacted.

[ Go to Contents ]

Measures of Child and Family Well-Being

The well-being measures address the first TANF goal of providing assistance to needy families so that children may be cared for in their own homes. They assess state performance in taking action to ensure that low-income working families continue to receive the supports they need so that they may provide food, health care, child care and other basic needs for their children. These measures also address the goal of ending the dependence of needy parents on government benefits by promoting job preparation and work, because in many cases, low-income parents need aid from government programs - particularly health care coverage, assistance in purchasing food, and support in paying for child care - in order to work. Assistance from these programs helps make it possible for families to move off welfare into employment and to progress on the job toward eventual full independence.

Measures of child and family well-being, while carefully monitored at the national level, have traditionally been difficult to measure at the state level. The Current Population Survey (CPS) is currently the best source of annual data for well-being measures at the national level, but its comparatively small sample sizes limit its ability to measure state-specific outcomes except in a few, large states. Various small area estimation techniques are currently used by the Census Bureau and others to produce reliable state-by-state estimates, including combining and averaging three or four years of data into "moving averages." Unfortunately, moving averages make it difficult to track improvement or declines in state performance over short periods of time.

If it is implemented as planned, the American Community Survey (ACS) would be the preferred data source for well-being measures. We anticipate that nationwide data appropriate to calculate state-by-state performance measures will be available for 2000. Once it is in full operation, the ACS (based on the decennial census long form) will be available every year for areas and population groups of 65,000 or more. It should be noted that use of any measure that relies on the ACS is contingent upon the continued availability of the new Census Bureau data. Appendix D includes descriptions of these and other national surveys.

Food Stamp Receipt

An important indicator of well-being is whether all members of a family have regular access to food and are free from hunger. Ideally, we would measure access to food directly, such as through the food insecurity and hunger scale developed by the U.S. Department of Agriculture. However, while this scale is administered annually as a supplement to the Current Population Survey, the sample sizes are not large enough to track changes at the state level. One possible alternative is to use an intermediate outcome measure, such as participation in the Food Stamp Program. The Food Stamp Program is an entitlement program that is available to help all low-income families who meet the national eligibility standards, including families receiving cash assistance and working families, to purchase food for an adequate diet. Several measures could potentially be used to gauge states' success in ensuring that eligible families receive Food Stamps. These include: (1) the percentage of families eligible for Food Stamps that receives them, (2) the percentage of poor children in working families who are receiving Food Stamps; and (3) the percentage of former TANF recipients receiving Food Stamps; as well as variations on these alternatives.

Under the High Performance Bonus, HHS has chosen to award bonuses based on a variation of option 2. Beginning in FY 2002, bonuses will be provided to the three states with the greatest percentage of low-income working households in the state receiving Food Stamps and to the seven states with the greatest percentage point improvement in the same measure. For this purpose, low-income working households would be defined as households with children under the age of 18 which have an income of less than 130 percent of poverty and earnings equal to at least half-time, full-year employment at minimum wage. The threshold of 130 percent of poverty was used because most, although not all, families at this income level are eligible for Food Stamps.

Measurement issues. While these measures are similar in that they attempt to capture the proportion of poor individuals receiving Food Stamps, they differ in the extent to which they can be influenced by the TANF program. The first option, the percentage of eligible families receiving Food Stamps, uses a relatively broad population as the denominator - the population that is eligible for Food Stamps. Although a large part of the Food Stamp caseload traditionally has received AFDC/TANF, this measure would address the overall effectiveness of the Food Stamp Program in reaching its target population, as well as the effectiveness of the TANF program in ensuring that individuals who are diverted from or leave cash assistance receive this benefit. The second measure, the percentage of poor children in working families receiving Food Stamps, is more narrow than the first in that it focuses on children in poor working families. The measure examines Food Stamp receipt by a group that up to now has had relatively low participation rates in Food Stamps, although the levels are increasing somewhat. The third measure focuses on whether individuals who had received TANF were receiving Food Stamps after they left the program. While this measure focuses on an outcome that can be directly influenced by the TANF program, it cannot capture the extent to which individuals who are diverted from TANF assistance on the front end receive Food Stamp benefits. Since Food Stamp eligibility is on a household basis, it should not matter much whether children or adults are the unit for the measure.

Data issues. The population measures are best measured by a combination of national survey data and Food Stamp administrative data. In terms of national survey data, the American Community Survey (ACS) is the preferred data source. These national surveys are best-suited to calculate the denominator of the first and second measures. The number of poor children (defined as "below the poverty line") in working families (or a reasonable proxy for working) would be measured directly using the ACS. Food Stamp eligibility would not be measured directly, but could be estimated based on the information collected, although some measurement errors will result from the different periods of observation (surveys collect annual data while Food Stamp eligibility is based on monthly income).

While the CPS and ACS collect data on Food Stamp receipt, surveys have historically under-reported information on receipt of public benefits and are not considered the best gauge of program participation levels. An alternative would be to use Food Stamp Quality Control (QC) data to calculate the numerator of the measures. Food Stamp QC data provide high quality, timely program enrollment data on an annual basis, although it is a sample and there are measurement differences between the QC database and national surveys (e.g., QC captures data on program participation and household circumstances including income for a specific month, while national surveys examine program participation and household circumstances over the past year).

The third measure - the percentage of former TANF recipients who receive Food Stamps - could be calculated by using linked TANF and Food Stamp administrative data. This would require states to use their TANF administrative data to determine those individuals who left TANF and to match this group against Food Stamp administrative data. This measure would be more workable if it was restricted to a limited period of time after TANF exit.

Fairness issues. In terms of treating states fairly and equitably, the size of the relatively broad population groups that form the denominators for the first two measures is likely to be influenced by factors that are beyond the reach of the TANF program. The number of families eligible for Food Stamps or in poor working families is likely to be affected by the economy in the state as well as other programs and policies the state may have in place - including child care, health insurance, state earned income tax credits, and others. States with more poor families would have to work with a greater proportion of families than states with fewer poor families.

In addition, all else equal, states with higher TANF benefit levels will probably fare better on the two population measures because many of their working poor families will be eligible for TANF as well as Food Stamps (participation rates for Food Stamps are higher when families are also receiving TANF). These states are likely to perform more poorly on the leavers measure because those families earning enough to exit TANF will have higher incomes, making them eligible for only a small amount of Food Stamp benefits, which might not be worth the effort needed to continue to participate. In states with lower TANF benefit levels, families with earnings would become ineligible for TANF sooner, making their Food Stamp benefit levels higher, and increasing the likelihood of their remaining on Food Stamps longer.⁽⁴⁾

Medicaid/SCHIP Receipt

The Medicaid and State Children's Health Insurance Programs (SCHIP) are designed to provide health insurance to low-income individuals. Medicaid provides coverage to adults and children;⁽⁵⁾  SCHIP covers children. State performance in ensuring receipt of Medicaid/SCHIP by eligible individuals can be measured in several ways. These potential measures are parallel to those suggested for Food Stamps and include: (1) the percentage of individuals eligible for Medicaid/SCHIP that receive the benefit, (2) the percentage of poor children in working families who are receiving Medicaid/SCHIP, and (3) the percentage of former TANF recipients receiving Medicaid/SCHIP.

Under the TANF High Performance Bonus, beginning in FY 2002, a bonus will be provided to the three states with the highest rates of Medicaid/SCHIP enrollment for former TANF recipients and to the seven states with the greatest percentage point improvement. Specifically, the measure of state performance will be the percentage of leavers who are enrolled in Medicaid or SCHIP four months after leaving TANF (and who are not currently receiving TANF assistance in that month). The population will be limited to leavers who were enrolled in Medicaid or SCHIP at the time of case closure. Virtually all such individuals should still be eligible for Medicaid or SCHIP four months later.

Measurement issues. The measurement issues for these Medicaid/SCHIP performance measures are similar to those for Food Stamps discussed above. While the three proposed measures are similar in that they attempt to capture the proportion of poor individuals receiving Medicaid/SCHIP, they differ in their relationship to TANF. The first and second measures use a relatively broad population as the denominator - the eligible population and working families with children, respectively. Thus, these two measures will capture the overall effectiveness of the Medicaid/SCHIP programs in reaching their intended populations, as well as the effectiveness of the TANF program in ensuring that individuals who are diverted from or leave cash assistance receive Medicaid/SCHIP. The third measure may be better focused on directly measuring the outcomes of the TANF program, but it does not capture the extent to which those who are diverted from TANF on the front end receive Medicaid/SCHIP.

The Medicaid/SCHIP participation rate might vary substantially depending on whether it was measured for adult leavers or for their children, because eligibility is at the individual level, not for the whole family unit. Data from HHS-funded studies of families exiting welfare indicate that in some states children are substantially more likely than their parents to be enrolled in Medicaid. (In other states there was little difference between adult and child participation rates.)

Data issues. These measures would be calculated in similar ways to those described above for Food Stamps. Medicaid administrative data - primarily the Medicaid Standard Information System (MSIS) - would have to be used to calculate the numerator of the rates because the American Community Survey (ACS) does not collect data on participation in Medicaid or SCHIP. All states have developed or are in the process of developing the capability of reporting Medicaid and SCHIP enrollment data using MSIS. This enrollment data would be collected and reported by states electronically on a quarterly basis and would be available within a reasonable period after the end of a quarter. Like the Food Stamp measures, national survey data, preferably the ACS, could be used to calculate the denominator of the first two rates - the population eligible for Medicaid and working families with incomes below the poverty line. The third measure would be calculated by matching the administrative records of individuals leaving TANF assistance with Medicaid/SCHIP enrollment data maintained on MSIS.

Fairness issues. Unlike Food Stamps, where eligibility is set at the federal level, states have some discretion over Medicaid and SCHIP eligibility. This creates two types of issues. On the first measure - which examines the proportion of the eligible population that receives Medicaid/SCHIP - states have very different eligible populations. A state with a broad eligible population may have more difficulty performing well on this measure than a state that defines Medicaid eligibility more narrowly. On the other hand, the second measure - which examines the proportion of poor working families that receives Medicaid/SCHIP - states that provide coverage to a broader segment of the population will do better because of their current policies. While the goal of this measure may be to encourage states to expand the reach of these programs, states that have already done so would start out with an advantage on the absolute measure if bonuses or penalties are based on level of coverage, and with a disadvantage on a measure based on improvements in coverage. Similar to measures of Food Stamp participation, some have argued that these measures are not appropriate since state policymaking on health insurance coverage to a broader eligible population goes beyond the purview of the TANF program.

As discussed above, states with higher TANF benefit levels will probably fare better on the two population measures because many of their working poor families will be eligible for TANF as well as Medicaid and participation rates for Medicaid are higher when families are also participating in TANF. In addition, for the two population measures, state administrators may not have control over the size of the needy population in their state. States with a relatively larger population below the poverty line may have more difficulty succeeding on these types of measures compared to states with a relatively smaller poor population.

Child Care Affordability and Quality

As more and more families move from welfare to work, there is an increasing need for access to affordable, high-quality child care. Affordable child care is necessary to enable most low-income parents, particularly single parents, to move from welfare to work. A growing body of research indicates that quality and stability of the child care setting influences outcomes for children as well as the ability of parents to retain employment. Moreover, high quality child care can contribute to the healthy development of children, especially children in low-income families who are often disadvantaged educationally as well as financially. However, most high-quality child care is unaffordable for low-income workers, including those moving from welfare-to-work, without government subsidies.

Measurement issues. There is no simple way to measure the success of state efforts to make high-quality child care accessible and affordable to low-income families. Under both TANF and the Child Care and Development Fund (CCDF), states have almost total flexibility to determine both eligibility for child care subsidies and the package of subsidies that is provided. An ideal measure of child care access and affordability would reflect the percentage of eligible children served and how great a share of their income all low-income parents are paying for child care, both those who are benefitting from subsidies and those who are not. However, such a measure could only be calculated by asking questions not currently included on any survey that produces annual state-level estimates. For the FY 2002 High Performance Bonus, HHS intends to use a measure which combines the fraction of eligible children receiving child care subsidies with the amount of the required copayment as compared to family income. These factors measure the breadth and the depth of state child care subsidies.

Measuring the quality of child care is even more complex. While research has indicated several factors that are believed to contribute to high-quality child care (e.g. training of child care providers, high staff-to-child ratios), there is no universal agreement on what constitutes high-quality child care. For this reason, in the FY 2003 High Performance Bonus HHS intends to use a process measure that indirectly assesses the quality of services by comparing actual rates paid to applicable market rates. The logic behind the selection of this measure is that families in states which reimburse at a higher fraction of market rates will have access to a broader range of child care options, including higher quality care, which often costs more than mediocre care. While high reimbursement rates do not guarantee that families will access high-quality care, low reimbursement rates ensure that low-income families will not be able to access such care.

Data issues. A direct assessment of the affordability and quality of child care used by low-income families would require a new survey effort, which is not feasible for an ongoing performance measurement system.

The measures used under the TANF High Performance Bonus can generally be calculated using a combination of administrative data reported through CCDF and Census Bureau information on family income. HHS measures access to child care based on the percentage of children in families with 85 percent or less of the state's median income (the maximum eligibility level allowed under federal law) who are served with child care subsidies. HHS measures affordability based on the relationship between the state's reported family co-payments for subsidized child care and reported family income. The final quality component of the High Performance Bonus measure of child care requires states that choose to compete for the Bonus to report additional data collected through their mandatory biennial market rate surveys.

Fairness issues. Given a fixed amount of funds to be spent on child care subsidies, states may choose to allocate it in a variety of ways. They may provide generous subsidies to a smaller number of families, or more limited subsidies to a greater number of families. If the spending per family is to be limited, this can be accomplished either by requiring families to pay a greater portion of the child care expense (reducing affordability) or by capping the amount they will pay per child (thereby restricting family choice of child care providers, and potentially reducing the quality of the care received). Additional funding can be used to expand a program in any of these dimensions. The High Performance Bonus measures combine these elements in order to reflect these tradeoffs and avoid promoting one choice over the others. However, because this measure has not been calculated in the past (and the precise details of the measure are still under development), it is unclear whether this measure will truly be neutral among all approaches.

Receipt of TANF and Other Types of Transitional Assistance

Another possible performance measure related to child and family well-being is a broad measure that examines the percentage of needy children that receives TANF and/or other types of assistance, including Medicaid/SCHIP, Food Stamps, and child care.

Measurement issues. In calculating this measure, care would have to be taken to ensure that individuals receiving benefits from more than one program are not double counted in the numerator of the rate. Otherwise, the extent to which states are serving eligible needy families would be overstated. This measure could be calculated based either on children or family units.

Data issues. This measure is relatively complex to calculate and would require a combination of both administrative and survey data. Similar to the Food Stamp and Medicaid/SCHIP measures, the denominator of the rate could be calculated relatively easily using the American Community Survey. However, because it requires knowledge of the receipt of a number of different programs, determining the numerator of the rate is more difficult. The ACS cannot be used to determine the numerator of the rate because it does not capture whether individuals use certain public assistance programs - particularly child care subsidies and Medicaid/SCHIP. While administrative data from each of these programs could be used to calculate the numerator, this is more complicated than other similar measures discussed in this paper because a number of administrative data sources would have to be used and linked together (TANF, Food Stamps, Medicaid/SCHIP, and child care). Thus, while it is not impossible to calculate this measure, it is more burdensome for states than some of the other related measures.

Fairness issues. States with more generous benefit programs for TANF, child care, and Medicaid/SCHIP will perform better on this measure - at least initially - than states with less generous programs. While the goal of this measure may be to encourage states to expand these programs, states that have already done so will start out with an advantage on the performance measure when capturing absolute performance. States that have more restrictive coverage could potentially perform better on a measure based on improvement. In addition, as discussed above, state administrators may not have control over the size of the needy population in their state. States with a greater population below the federal poverty line may have more difficulty succeeding on this type of measure compared to states with fewer poor people.

Extreme Poverty Rate

Another alternative approach to measuring the well-being of children and families is to look directly at income levels, rather than program participation. One proposed measure would examine the rate of extreme child poverty - those living at 50 percent of the poverty level or less - or the change in this rate. This measure would focus on the extent to which states have success in improving the status of a very needy group - the "poorest of poor." At the July 1999 consultation meeting, there was a general sense that a measure below 100 percent of poverty was more likely to be influenced by changes in TANF policy than the standard measure at 100 percent of poverty, because most states' cash assistance need standards and payment levels are substantially below the official poverty line.⁽⁶⁾  In addition, these very poor families may be at greatest risk of experiencing substantial material hardships, such as hunger or homelessness.

Measurement issues. The simplest way to measure extreme poverty would be using the standard definition of income, which includes cash assistance, but not the cash equivalent value of food stamps, or the effects of taxes. A modified definition of income, which includes both taxes and transfers, would better capture the effects of a range of state policy choices on family well-being. The extreme poverty rate could be measured based on family units, on individuals, or on children. Focusing on children gives more weight to the circumstances of large families ( i.e., each child in a large family would be counted separately) who are somewhat more likely to be poor. By including all families, whether or not they receive TANF assistance, and whether or not they are employed, this measure rewards states that provide a comprehensive safety net for the most disadvantaged.

Data issues. The preferred data source for this measure is the ACS, which will provide reliable, timely state-specific data that could be used to calculate both the numerator and denominator of the rate, starting in 2000. The CPS provides data that could be used to calculate this measure in the short-term, but for all but the largest states we would have to rely on small area estimation techniques, such as moving averages (combining and averaging three or four years of data).

Fairness issues. The extreme poverty rate is likely to be affected by a number of factors - some of which are outside the scope of a state's TANF program. For example, the poverty rate is likely to be affected by a state's economy and by the policies of other social programs for the poor. Both the absolute rate and the change in the rate would be affected by these other factors. However, using the change in the extreme poverty rate as a measure might give states with historically high poverty rates an incentive to improve their performance, since they have a greater chance of performing well on a measure based on improvement.

[ Go to Contents ]

Family Formation/Stability Measures

Family formation/stability measures address another goal of the TANF program - to promote marriage and encourage the formation and maintenance of two-parent families. This goal is based on the belief that families are one of the strongest factors in developing and sustaining high levels of individual competence and functioning in our complex society. The number of parents living with a child is generally tied to the amount and quality of human and economic resources available to that child. Children who live in a household with one parent are five times more likely to have family incomes below the poverty line than are children who grow up in a household with two parents.

Percentage of Children Living in Married-Couple Homes

Measuring the percentage of children living in married-couple families addresses the TANF goal of encouraging the formation and maintenance of two-parent families. Unlike a measure of nonmarital births, this measure considers whether parents marry and stay married for as long as they have children at home, not just at the time the child is born. In addition, it addresses the well-being of low-income children and families and could serve to stimulate state interest in a range of strategies that promote intact families. The TANF High Performance Bonus includes a measure of the percentage point change in the rate of all children who reside in married-couple families, based on a comparison of data between years.

Measurement issues. The primary issue for this measure is the population to which it will be applied. Under the Notice of Proposed Rulemaking (NPRM) for the TANF High Performance Bonus, HHS proposed to apply this measure to children in families with incomes below 200 percent of poverty who reside in married-couple families. HHS proposed to restrict the measure to poor and near-poor families because these are the ones who are most likely to be affected by welfare policy and to be targeted in state programs to promote marriage.

However, many commentators expressed concern about this measure, noting that if the measure was focused on poor or near-poor children, as the NPRM proposed, there was the potential for states to be rewarded for undesirable outcomes. For example, an increase in the number of low-income families, such as might be caused by an economic downturn, would also likely result in an increase in the share of low-income two-parent families. Conversely, policies that promoted marriage or rewarded work among two-parent families might cause these families to have incomes greater than 200 percent of the poverty level, causing a state to be punished for a positive outcome. In response to these concerns, in the Final Rule for the High Performance Bonus, HHS adopted a broader measure rewarding those states with the greatest increase in the percentage of children who reside in married-couple families, regardless of family income.

Data issues. Using this measure would entail no new data collection responsibilities on the part of the state. It is anticipated that national data will be available to measure state performance from the Census Bureau's decennial and annual (e.g., ACS) demographic programs. The Census Bureau's decennial and annual demographic programs would provide uniform, objective, and reliable state-level data beginning in 2001 (with respect to 2000). As noted above, use of this (or any measure that relies on the ACS) is contingent upon the continued availability of the new Census Bureau data.

Fairness issues. Since this is a population-based measure, the TANF agency would have limited ability to influence this family formation measure beyond the TANF population. However, the TANF goal of supporting two-parent families is not limited to "needy families," so it might be appropriate for TANF agencies to look beyond their traditional service population of families receiving cash assistance.

Out-of-Wedlock Birth Rate for TANF Families

This measure would report on the rate of nonmarital births that occur to TANF families and address the TANF goal of preventing and reducing the incidence of out-of-wedlock pregnancies. This measure differs from the existing Bonus for Reduction in Out-of-Wedlock Births, which measures changes in nonmarital birth rates for all families in a state.

Measurement issues. The advantage of restricting a measure of nonmarital births to TANF recipients (rather than all families) is that TANF agencies have much more ability to affect the choices of TANF recipients, through financial incentives, life skills classes, parenting education, and referrals to family planning providers, than they do to affect the general population.

However, such a measure has the possibility of undesired consequences. There is already some controversy over the "family cap" policies that several states have already adopted, which deny additional benefits to families for children conceived while the family was receiving welfare. There would probably be greater concern if states included a requirement not to have additional children in a personal responsibility contract, and denied assistance entirely to recipients who fail to comply. Policy makers might wish to add to this measure restrictions to ensure that the desired goals were not achieved through unacceptable means; for example, the Bonus for Reduction in Out-of-Wedlock Births includes a requirement that states which receive the bonus must also have achieved a reduction in their abortion rate.

Another concern is that the composition of the denominator of the measure - the TANF caseload - will be constantly changing. For example, it is widely believed that as the welfare caseloads have declined, the most advantaged (job-ready) recipients have disproportionately left the rolls, leaving a higher proportion of more disadvantaged, harder to employ, families behind. It is possible that the least job-ready recipients may have a higher rate of nonmarital births than the more job-ready individuals who leave TANF more quickly. Thus, it may be difficult to distinguish whether a change in the nonmarital birth rate was due to a change in the number of out-of-wedlock births or a change in the composition of the TANF caseload.

Data issues. There are two potential data sources for calculating the out-of-wedlock birth rate for TANF families. The most accessible and timely data source is TANF state administrative records. As part of their aggregate data collection for TANF, states are required to report on a quarterly basis the monthly number of families receiving TANF with an out-of-wedlock birth. The quality of these data has not yet been assessed. The National Center for Health Statistics (NCHS) is another potential data source. States report data on all out-of-wedlock births in their state to NCHS on an annual basis. These data could be matched to TANF administrative records to determine the out-of-wedlock birth rate for TANF families. However, this type of match has not been done yet, so its feasibility and burden would have to be determined.

In addition, a few states do not directly ask the mother's marital status on the birth registration form, but instead infer it from other information collected (typically, whether the mother and father have the same last name). This reduces the comparability of data among states.

Fairness issues. This measure has not previously been calculated. However, it is known from the existing Bonus for Reduction in Out-of-Wedlock Births, that there is a great deal of variation across states in the percentage of all births which are out-of-wedlock. This variation appears to be related to the demographic characteristics of the state's population, rather than to specific state policies. It is likely that this variation will also occur among states with respect to nonmarital births in the TANF population, again for reasons other than state policy choices. This suggests that a measure of improvement (as used under the Bonus for Reduction in Out-of-Wedlock Births) would be more appropriate than a simple absolute rate. However, as discussed previously, improvement is generally more difficult to achieve for states which have already achieved a high level of performance (in this case, a low rate of nonmarital births).

Endnotes

4.  Unlike TANF, Food Stamp income eligibility standards and benefit levels are set at the national level. Eligibility is based on the number of people in the household and the amount of income the household has; most households must have income at or below the Federal Poverty guidelines after deductions are allowed. Almost all types of income, including cash assistance, are counted to determine if a household is eligible. Food Stamp benefits are 100% federally funded. [return to text]

5.  Medicaid is a federal-state matching entitlement program. The federal matching rate is inversely related to a state's per capita income, and can range from 50 to 83 percent. Within federal guidelines (states are required to serve some population groups and are permitted to serve others), each state designs and administers its own program. Thus, there is substantial variation among states in coverage, types and scope of benefits offered, and amounts of payments for services. Applicants' income and assets must be within program financial standards - for some population groups these standards vary among states; for others, standards are set by federal law. [return to text]

6.  A separate provision of PRWORA requires the chief executive officer of each state to report annually on the child poverty rate in the state. If, as a result of PRWORA, the child poverty rate of the state has increased by 5 percent or more, the state must prepare and submit a corrective action plan outlining the manner in which the state will reduce the child poverty rate. [return to text]

Where to?

Top of page
Contents

Main page of report
Contents of report

Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation (ASPE)
Administration for Children and Families (ACF)
U.S. Department of Health and Human Services (HHS)

Updated: 0212/01