ReWORKing Welfare Technical Assistance for States and Localities. 13. Evaluating Program Performance


How will you monitor the implementation of your work first program and measure its success? A variety of mechanisms are available to measure program performance. Outcome and process measures are useful management tools that can focus staff effort and identify weak areas. Formal evaluations can measure program impacts.

The performance measures you choose should reflect and support the program's goals and objectives. (See section 10, on tailoring work first to the program's goals.) Be careful that the measures promote the outcomes you want and do not neglect important areas or inadvertently encourage undesired outcomes. In addition, be aware that different measures-or the same information measured in different ways or at different points-will yield different information. Finally, the data required for some measures may be costly or difficult to collect. In choosing performance measures, bear in mind what information you already have and what your management information system can be modified to do. Make sure that the data can be collected relatively easily and accurately, and that data collection will not be burdensome in terms of cost or staff time. Also, bear in mind any data-gathering requirements mandated at the federal and state levels for other purposes, and try to coordinate them with your own needs.

It is a challenge to develop a performance measurement system that provides meaningful and practical information and that guards against creaming and inflated outcomes. Such a system requires a long-term investment to design, implement, adjust, and maintain. The challenge becomes even greater if the work first program is being operated in coordination with other agencies.

Outcome and Process Measures

  • Job placements. The traditional outcome measure in a work first program is the number of participants who get jobs. It is also useful to know the length of time individuals participated in the program before finding employment, the point in the program flow at which they gained employment, and their wage levels and number of work hours. Follow up to learn how many are still working after 30 days, 90 days, or longer.

Be aware, however, that measuring only job placements can result in creaming (if staff focus on the most employable participants) or in inflation of program outcomes (if staff report placements that are unconfirmed or do not last). An overemphasis on placement can also inadvertently encourage staff to neglect, defer, or sanction the hardest-to-serve participants. Measuring participation or coverage (see below), in addition to placements, can help guard against this. You might also want to look at the characteristics of those who do not get jobs, to see if additional focus on some groups of participants may be needed.

  • Welfare leaving and returning. By changing the message and culture of welfare, a work first program can have an effect beyond that captured by measuring job placement. It can therefore be useful to look more broadly at the number of people who leave welfare for work, even though you may not be able to distinguish between those who leave as a result of the program and those who would have found work anyway. It is also important to monitor and measure recidivism, so that you know how many of those who leave return later. If your program includes applicant job search or other up-front diversion policies, you will want to measure the number of applicants diverted from entering the welfare rolls. Again, monitor to see if they are reapplying later.
  • Participation. Bringing participants into a work first program is an important operational challenge. Participation measures highlight the importance of this challenge and focus the attention of administrators and staff on meeting it. They can also be useful in identifying bottlenecks and other operational problems or weak program areas. However, too strong an emphasis on participation measures can divert the focus from the real goal-employment. Moreover, if the definition of participation is a narrow one, you may understate the number of people who are appropriately involved in the program mandate.

Because participation measures are ratios, it makes a great deal of difference what is included in the numerator and in the denominator. In establishing the numerator, program planners need to decide what activities and statuses to count as participation. The denominator can be the entire welfare caseload, those who are mandatory for the program, or those who actually enroll. Each of these denominators will show successively higher rates of coverage but will omit certain groups. (Note that federal law specifies the definition of participation for the purpose of meeting federal requirements; see Appendix A.)

  • Coverage. Coverage is a broad definition of participation that includes all those who are considered to be in an acceptable status. For example, coverage might include those in program activities as well as those who combine work and welfare, even if they are considered exempt from work first. Coverage might also include activities such as counseling or drug treatment where appropriate. Some programs might include sanctioned cases, recognizing that those cases are being appropriately dealt with under program guidelines. Others may choose not to include sanctioned cases in the coverage rate, as a way of encouraging program staff to work harder to bring people into compliance.

By measuring coverage rather than participation, you focus attention on those individuals who are left unserved by the program and aim to minimize their number. This protects against creaming by encouraging staff to work with-and giving them credit for working with-everyone.

  • Other measures. In general, you will want to highlight one or two performance measures that reflect the program's primary goals. However, it is useful to track a variety of other measures that can provide additional clues to program performance. These can furnish information on intermediate outcomes and implementation success. For example, completion rates and length of time in education and training activities can give you an idea of whether those activities have successfully adopted the quick-employment philosophy of work first. Other measures can also reflect how the program is doing at achieving additional or secondary goals. For example, you may want to measure average family income if reducing poverty is a goal. You may also want to consider ways to measure the stability and appropriateness of child care placements. Finally, other measures can provide a more complete picture of your program, to ensure that no unintended consequences have resulted from program policies or other performance measures.

Program Evaluation

Welfare recipients regularly find jobs and leave the rolls even without welfare-to-work programs. Because the caseload is so dynamic-and so sensitive to economic and other factors-it is difficult to judge the success of your program without knowing what would have happened in its absence. Random-assignment evaluations answer that question, by separating the effects of the program from changes that would have occurred anyway. Potential participants are randomly assigned either to the program or to a control group that is not eligible for program services. The characteristics of the two groups are therefore comparable, and both groups face the same labor market and other conditions. Subsequent differences in employment, earnings, or welfare receipt between the two groups can then be confidently attributed to the program.

Table 2 illustrates the difference between outcomes and impacts. As the table shows, program A clearly has the best outcome of the three, with a 70 percent employment rate, but-perhaps because of a strong economy or because participants were highly skilled-most participants would have found jobs even without the program's help. Program C has the smallest outcome but the largest impact; it was the most successful of the three at increasing the number of people who found jobs. The table illustrates the point that strong outcomes do not necessarily mean that a program is working well. Random-assignment evaluations can uncover a program's true impact.

Program planners may also wish to conduct internal random assignment for pilot projects to test different approaches before implementing them programwide. For example, if you are unsure whether it is worth allocating resources for applicant job search, you might randomly assign new welfare applicants either to applicant job search or to a regular route into the program. You could then compare welfare approval and employment rates for the two groups to determine whether you want to go ahead with that program component. The same technique could work for post-placement services, job development, and other activities.

Table 2

Outcomes Versus Impacts

               Outcome:                Control:                   Impact:         
       Employment Rate    What the Employment Rate     The Difference the    
       with the Program   Would Have Been Without       Program Made       
Program                    the Program

    A             70%                    65%               5 percentage points    

    B             50%                    40%               10 percentage points   

    C             40%                    20%               20 percentage points