Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations. c. Source of Baseline Data

10/01/1996

Good baseline data can be collected either from administrative records or from special forms or surveys, if sufficient planning and resources are devoted to the effort.

Administrative Records. Administrative records are the best source for historical data on outcome variables, especially if the state maintains these records in a consistent format over time. Unemployment Insurance (UI) records data on employment and earnings generally are available. States vary in the quality of these data and in archiving procedures, however, so it may be difficult to obtain these data retrospectively.(2) Administrative data from the AFDC program and related programs from before random assignment generally will be available for ongoing cases, and may be traceable for applicants who participated in the past. Administrative data are less attractive sources for basic demographic data, however. This is because for cases with no previous AFDC history, data as of the end of the month or quarter after random assignment (often, but not always, as entered at application) usually are the only data available. Furthermore, the baseline data entered into the administrative system may be of poorer quality for applicants who are not approved for assistance (if the data are there at all). Administrative data usually are the sources for key identifiers such as social security numbers, but such data again are likely to be of higher quality for approved applicant cases than for denied cases (if data on denied cases are tracked at all); thus, the quality may differ between experimental and control group cases. Finally, administrative data generally are poor sources of contact information.

In principle, administrative systems may be modified to address some of these problems. For example, systems can be modified to record information on denied applicants or to keep certain background variables as recorded at the time of random assignment. Still, such data must be entered by staff for whom they are not immediately useful, and who might also be learning new procedures.

Surveys or Special Forms. Surveys or special forms can be attractive for baseline data collection because they allow collection of information that is not usually in automated data systems. In addition, if timed appropriately, they can be used to obtain data on denied applicant cases. Such special data collection efforts are expensive, but so are modifications to large automated systems. In general, the most useful strategy for collection of baseline data is to collect such data in the program office immediately before random assignment (either in a brief interview with an intake worker or a staff member from the evaluation contractor or through a paper form filled out by the sample member) and then to have staff members review the data for completeness and accuracy.

A telephone survey just after random assignment is a less desirable strategy. Even if sample information is sent to the evaluator quickly, there is often a lag of several months between when the survey begins and when the sample member is located. Surveys several months after random assignment run the risk of lower response rates, contamination by the intervention, and different response rates for experimental and control group members; consequently, they do not provide true baseline data. If the evaluator is not yet chosen and the survey is not yet designed when random assignment begins, there will be further lags before data collection can begin.

If the major reason to do a baseline survey is to obtain contact information for a follow-up survey, and a contact information form was not filled out at application/redetermination, a postcard sent to research sample cases may be acceptable. However, this approach also runs the risk of contamination if there is a differential response rate for treatment and control group members. A small incentive payment to sample members for returning the postcard may help avoid such differences.