How Effective Are Different Welfare-to-Work Approaches? Five-Year Adult and Child Impacts for Eleven Programs. Measurement Issues


The measures of child well-being analyzed in this section were constructed from mothers' responses to the Five-Year Client Survey. Respondents were asked about their children's academic functioning, health and safety, and other outcomes. (See Table 11.1.) Respondents answered for their own children  whether biological children, legally adopted children, or stepchildren. Although they were asked about all children regardless of age, this analysis is limited to children who were aged 18 or under at random assignment. This section provides a discussion about the construction of these measures. (Details about these measures can also be found in Appendix J.)

Respondents were asked a variety of questions about their children, concerning events or situations since a focal date. For most respondents the focal date was the date of the two-year interview. For those who did not answer the two-year survey, the focal date was the date of random assignment. Thus, for most respondents, child outcomes are measured for years 3 to 5 of follow-up (that is, since the two-year interview point). Some outcomes, such as dropping out of school, were not restricted to the period of time since the focal date. Instead these outcomes were measured "ever" in a child's life.

The first group of measures captures children's academic functioning including whether or not they had been suspended or expelled from school, repeated a grade, dropped out of school, or attended a special class or special school for a physical, emotional, or mental condition.

The second group of measures captures children's health and safety, including whether they had a physical, emotional, or mental condition that required frequent medical attention, frequent use of medication, or the use of any special equipment; whether they ever had a physical, emotional, or mental condition that demanded a lot of attention and made it difficult for the respondent to attend work or school; and whether they had an accident, injury, or poisoning that required a visit to a hospital emergency room or clinic. This last outcome is only a rough proxy for child safety. On the one hand, it may measure neglect if children are experiencing more accidents or injuries as mothers increase their work effort and children are left unsupervised, with abusive partners or other adults, or, possibly, placed in unsafe child care arrangements. On the other hand, this outcome may simply reflect a mother's ability to purchase medical care. Emergency room use may be used as a replacement for visits to a doctor or a clinic.

The final group of measures concerns living arrangements (whether a child did not live with their mother because she could not care for them) and teenage parents (whether a child gave birth as a teenager). In this analysis children were considered to have been teenage parents if they had a child at age 18 or under during the five-year follow-up period. Whether children did not live with their mother because she could not care for them may have numerous interpretations: It may capture the consequence or result of a government intervention in the family, for example, being forced to place children in foster care. However, it is also possible that a mother voluntarily elected to place her children in another living arrangement, which may provide a better environment. This outcome may be also interpreted more generally as a measure of household composition or living arrangements rather than a direct measure of child well-being.

Outcomes in this chapter are presented for any child in the family and for each child in the family. Measures presented for any child indicate the percentage of families in which at least one child experienced a certain outcome (for example, "any child ever repeated a grade" indicates that at least one child in the family repeated a grade) and provide a general snapshot of child outcomes at the family level. These measures are similar to those analyzed at the two-year follow-up point. Measures presented for each child in the family indicate the percentage of children who have experienced a certain outcome. Thus, unlike most of the information collected in the survey, these outcomes are specific to a child within a family. Each child in a family who was 18 or under at random assignment is represented in the impact analysis.(11) The 5,342 families in this analysis had a total of 13,726 children.

Four age groups of children are examined:

  1. Children who were toddlers at random assignment (aged 6 and 7 at the five-year follow-up; 7 percent were age 8);(12)
  2. Preschool-age children (8 to 10 at the five-year follow-up; 4 percent were age 11);
  3. Young school-age children (aged 11 to 14 at the five-year follow-up; 3 percent were age 15); and
  4. Adolescents (aged 15 to 23 at the five-year follow-up; 78 percent were age 15 to 20; 22 percent were 21 to 23, and 1 percent were 24).

The ages of respondent's children varied across sites. As discussed in Chapter 1, mothers with children aged 3 or over were required to participate in most sites. However, in Grand Rapids and Portland, the participation mandate was extended to single mothers with a child as young as age 1. Although this analysis treats children mutually exclusively, the mothers of these children within and across each age group are not mutually exclusive: 41.5 percent of respondents have more than one child within one age group and 49.6 percent have at least one child in more than one age group. The sample size for each of these age categories and the general structure of how the samples were derived for these two chapters are presented in Figure 11.1.

Although the outcomes covered in this chapter provide important information about child well-being, they have some limitations. First, all of them are based on mothers' reports, which may differ from teachers' or children's reports or from direct assessments of the children.(13) Second, the outcomes provide only a snapshot of particular domains of children's development. For example, children's problem behavior (such as their expressions of anxiety, depression, or aggression) and positive behavior (such as their interaction with peers and others) are not captured, and it may be that the NEWWS programs are most likely to affect these behaviors. Measures collected in the Child Outcomes Study described in Chapter 12 address these limitations.

Third, similar measures were collected and constructed for each child regardless of age, yet these measures have different implications by child age. For example, partly because of age requirements for employment, an adolescent who repeats a grade may be much more likely to drop out of school and possibly enter the labor force than a younger school-age child who repeats a grade. Dropping out of school is highly correlated with future labor force participation.(14) In addition, control group levels on these outcomes might differ by age group: Control group levels of suspension or expulsion are naturally higher for adolescents than for early school-age children, making it harder for programs to produce any statistically significant changes.

To provide some basis for evaluating the magnitude of impacts, Tables 11.2-11.6 and Tables 12.1-12.5 report effect sizes in the last column. The accompanying text box describes effect sizes in more detail.


Effect Sizes and the Magnitude of Effects on Child Outcomes

Evaluating the effects of welfare-to-work programs on child outcomes also requires an assessment of whether the effects are big or small. An impact may be statistically significant, but is it large enough to be deemed important? Evaluating the size of an impact on various measures of adult economic outcomes is relatively straightforward. For example, most can assess whether or not an impact of $200 is a big or small effect on an individual's annual income. It is much more challenging to evaluate whether or not a 10 point change in a scale measuring a child's behavioral problems or a 5 percent change in a scale measuring school progress is big or small.

One method of assessing impact size is to standardize it. To do this, impact estimates can be converted into effect sizes. Effect sizes are computed by dividing the impact (the difference in outcomes between the program group and the control group) by the standard deviation of the outcome for the control group. The value of the effect size provides a standardized measure of the program impact that can be used to compare program impacts on outcomes with very different scales. Effect sizes generally range from 0 to 1; a larger absolute value indicates a larger impact on an outcome and a smaller absolute value indicates a smaller impact. Effect sizes rather than percentage changes are reported in the last column of Tables 11.2-11.6 and Tables 12.1-12.5.

How large are these effects? Generally effect sizes of 0.1, 0.3, and 0.5 are considered small, medium and large, respectively.a However, these benchmarks are based on nonexperimental studies that cover a broad range of topics. One method is to compare effect sizes on adult economic outcomes and effects sizes on child outcomes. Most welfare and employment programs generate effect sizes of about 0.2 to 0.3 on outcomes such as employment and earnings, and effect sizes on child outcomes are generally half this size. Another method is to compare these effect sizes with effects produced from child-focused interventions such as the Perry Preschool Program and the Abecedarian Project. These child-focused interventions produced effects that generally ranged from 0.2 to 1.0. Finally, it is important to consider that even small effects may have a large impact on the future well-being of a child. Longitudinal studies of children have found that achievement and behavior problems can have important implications for children's well-being as adults.b For example, achievement and problem behavior in early childhood are related to adolescent achievement and behavior. Small effects (for example, 0.1-0.2) may continue to have implications for children over their lives.

a Cohen, 1988; Lipsey, 1990.
b Caspi et al., 1998; Masten et al., 1995.