Paper presented at the
National Association for Welfare
Research and Statistics (NAWRS)
39th Annual Workshop in Cleveland, Ohio
August 11, 1999.
By
Julia B. Isaacs
Director, Division of Data and Technical Analysis
Office of the Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)
http://aspe.hhs.gov/hsp/index.htm
The views expressed in this paper are those of the author
and should not be construed as representing the views of the
Department of Health and Human Services, or any office therein.
With welfare caseloads dropping dramatically in the past four years, there has been increased public interest in an apparently simple question: what is happening to families who leave welfare? Do they get jobs? What is their economic status? How are their children faring? To answer these questions, the Department of Health and Human Services (HHS) awarded grants to 14 states and large counties to track and monitor outcomes for families leaving welfare.(1) The purpose of this paper, and of this panel presentation, is to examine the survey instruments used by these fourteen grantees in their studies of welfare “leavers.”
In this first paper, I examine the instruments from a crossproject perspective. After an initial overview of the 14 projects and the research topics they address, the majority of the paper examines the actual wording of survey items across eleven different instruments. A sample of only five selected outcomes — food stamp receipt, health insurance coverage, food insecurity, health status/access to health care, and knowledge of Medicaid and food stamp eligibility — were selected for this in-depth analysis.(2) Additional outcomes — employment and earnings, barriers to employment, the financial situation of those not employed, and child wellbeing — are discussed by the other co-presenters in this panel, although not from a crossproject perspective.
The crossproject review of instruments in this paper may serve different functions for different audiences. Researchers who are currently designing surveys of leavers may pick up some interesting examples of survey items and reflections on the differences caused by variations in wording of items. For the grantee researchers who are in the midst of analyzing results from the field, the review may provide some aid in “benchmarking” their survey findings to findings in other localities. That is, if a state finds that X percent of its leavers answered a certain question, it is important to know whether any other state or national surveys asked the same question, so that the state’s results can be compared to some benchmark. Analysts who are interested in examining findings across studies can benefit from a better understanding of the comparability of different instruments. Apparent differences between State A and State B may be partially caused by differences in the design of survey items.
Those anxious to know more about the wellbeing of former recipients may be disappointed because this review does not report actual findings. It does, however, provide some preview of the types of information that will be available over the next year, and thus some indication of the extent to which states are asking questions that will provide information needed for policy decisions. Finally, those in HHS, USDA, and other organizations that have provided sample survey items or technical guidance to the grantees may benefit from learning the degree to which researchers have incorporated various survey items in their instruments.
All 14 of the studies funded by the Department of Health and Human Services, Office of Assistant Secretary for Planning and Evaluation (ASPE), are using a combination of linked administrative data sets and surveys of former recipients to monitor the economic status and general wellbeing of families leaving welfare. At this point (July 1999), most grantees are about half way through their projects; and six have released interim reports based on administrative analyses of an early cohort of leavers. Most grantees define this early cohort as those that ceased receiving cash assistance in late 1996 or early 1997, as states were beginning to implement the Personal Responsibility and Work Opportunities Act (PRWORA). Grantees are still working on their administrative analyses of later cohorts of leavers (e.g., those leaving in late 1997, 1998 or early 1999), and on collecting and analyzing survey data for these later cohorts. Although a number of states have released reports based on statefunded surveys of leavers, most of the HHS-funded surveys are still in the field. Final reports, which will combine analyses of linked administrative data with findings from surveys of former recipients, are scheduled for completion at various times over the next twelve months, depending on the schedule of the individual grantees.
As stated above, each grantee is fielding a survey to gather information about the overall wellbeing of families leaving welfare. Although research proposals ranged across grantees, most grantees proposed surveying a sample of between 600 and 1200 former recipients, at a point between 4 to 12 months after exit from cash assistance.(3) With one exception, grantees are conducting mixed mode surveys, consisting of telephone interviews for the majority of households, supplemented by in-person follow-up for those who cannot be reached by telephone. Most surveys are expected to take 20-40 minutes. In this short time period, grantees are planning to ask questions across a broad array of topics, including employment and earnings, program participation, health insurance, child wellbeing, barriers to selfsufficiency, etc.
One key challenge has been the design of a survey instrument to gather information across this broad array of topics, within the constraints of a timelimited telephone interview. One potential solution to this challenge would have been to design a single survey for use across multiple sites. Such an approach would have had the advantage of providing common measures for crossstate comparisons and the development of a national picture of welfare leavers.(4) It was rejected early in the design of the project, however, because of its disadvantages, particularly in the current era of devolution of welfare policy making to the state and local levels.
One disadvantage of a nationally designed survey is that it might not address questions of most policy relevance to a particular state. For example, a state with a fullfamily sanction might want to ask different questions than a state with a partial sanction. Second, a national study would not be as effective in building state and local capacity for ongoing TANFrelated research to support state and local policy development. Finally, the consensus process for working with multiple states to develop a single instrument would likely be quite lengthy, with additional time required for official clearance of a national survey instrument by the Office of Management and Budget (OMB). A lengthy research process, however, conflicted with the desire of both Congress and state and local policy makers to know as soon as possible what was happening to families leaving welfare.
In sum, the Department of Health and Human Services decided to award grants that provided each research team with considerable flexibility in designing its own research project, including its own survey instrument. At the same time, Federal staff worked to encourage some degree of comparability of outcomes across states, within the constraints of a tight time schedule and limited resources. Steps taken to facilitate comparability included: holding an initial planning meeting in November 1998; establishing of an electronic “listserve” for discussion among researchers; developing consensus on a common definition of the “leavers” study population;(5) developing guidance on a proposed set of nine commonly reported administrative outcomes for leaver studies;(6) and, of most relevance for this paper, disseminating resources on surveys, including examples of survey items.
As shown in Table I , grantees were provided with paper copies of, or website addresses for, selected components of the Census Bureau’s Survey on Income and Program Participation (SIPP) and Survey on Program Dynamics (SPD), the USDA’s “Food Security/Hunger Core Module,” the Urban Institute’s “National Survey of America’s Families (NSAF), and other selected survey items. Some of these items were distributed in a resource book provided to all participants at the first grantee meeting, others were the subject of presentations at that same meeting, mailed to all grantees, or discussed over the listserve, primarily in December 1998 and January 1999.
Given the nature of the grants, and the limited amounts of technical assistance provided on surveys, one might expect that each grantee would develop quite different surveys. On the other hand, there was sufficient sharing of questionnaires and discussions on the listserve that one might expect some areas of similarity. Most of the rest of this paper reviews specific items to get some sense of the similarity and dissimilarity in the grantees’ survey instruments. Before doing so, however, I provide a brief overview of the types of research topics included in the grantees’ study designs.
In their research plans, all grantees proposed to collect survey information about employment and earnings, household income, program participation, and child and family wellbeing. There were differences in emphasis, however, as some grantees proposed asking extensive questions about employment and training services, others focused on collecting detailed information on household income across all household members, and still others wanted to ask extended questions on child wellbeing. A brief overview of the questions addressed in the survey instruments is provided below, with questions grouped into nine topical areas for ease of presentation.(7)
Whereas all fourteen grantees were included in the overview of research topics and Figures 1-9, a total of 11 instruments are included in the tables and analyses in the remainder of this paper. There are only 11 instruments in this review because two grantees — Cuyahoga County and Los Angeles County — are sharing the same survey instrument,(8) and two grantees — Massachusetts and New York — were still in the process of drafting instruments as of July 1999. Of the eleven instruments, six are clearly written as Computer Assisted Telephone Interviews (CATI) and the other five do not appear to be so, at least not from the versions shared with the author. An average interview length is provided for half the surveys; these averages range from 15 minutes (South Carolina) to 45 minutes (Missouri), with others at 20-30 minutes, 35-40 minutes, and 40 minutes. An annotated list of the eleven survey instruments is provided in Appendix I.
Most of the 11 survey instruments have 6 to 12 sections, spanning the breadth of topics discussed above. For this analysis, I identified all questions which address the following outcomes: food stamp receipt; health insurance (including Medicaid receipt); food insecurity; access to health care and health status; and knowledge of transitional benefits. Each of these five outcomes is discussed in turn below, with Tables II – VI providing summary information about each outcome, and Appendices II – VI providing actual wording of survey items for these five areas.
Ten of the eleven instruments ask questions about food stamp receipt. Five of these simply ask one question, such as “does your family currently receive food stamps?” and five ask additional questions, including whether food stamps continued after the household left cash assistance; if not, why not; and the amount of food stamps received. Illinois asks two additional questions not seen on other surveys: whether or not the former recipient has applied for food stamps since leaving TANF, and if not, why not. The Georgia survey does not appear to ask any questions about food stamp receipt, presumably relying on administrative records for this information. Illinois and Georgia are shown in the first and last columns, respectively, of Table II-A, which shows the types of questions asked by each grantee.
The exact phrasing of questions about food stamp receipt varies across three dimensions:
Though these phrasing variations may not make a large difference, one might expect slightly larger measures of receipt when measured over a sixmonth as opposed to a onemonth period, and larger measures of receipt when including food stamps received by any member of the home or household.
In addition to questions about food stamp receipt, eight instruments include questions about the Women, Infants, and Children (WIC) Supplemental Nutrition program and six ask about children’s receipt of school lunches (see Table II-B). A few also ask questions about the school breakfast or summer feeding programs. In addition, though not shown in Table II-B, ten of the eleven instruments ask questions about nongovernmental assistance, such as meals or food from shelters, food kitchens, or food banks (see Appendix IV-B).
Grantees ask many more questions about health insurance coverage than about food stamp receipt, reflecting the greater complexity of the subject. As shown in Table III, grantees are asking between five and ten questions about health insurance, with the questions falling into five general areas:
Most of these health insurance questions are asked twice, both with regard to the leaver herself, and with regard to her children (denoted with a “C” in Table III). Though not shown in the table, one survey (Cuyahoga/L.A.) also asks about coverage for the leaver’s spouse/partner, and another (Wisconsin) asks all health insurance questions about each member of the household, allowing different responses for various adults and children (see Appendix III for detailed wording of health insurance questions).
The wording of questions about Medicaid receipt may be quite important, given the fact that Medicaid coverage is a less tangible benefit than receipt of a monthly cash grant or a monthly food stamp allotment. Medicaid receipt may be under reported, if a recipient does not think of herself as “receiving Medicaid benefits” in months where no one in the family uses the health care system or if a Medicaidpaid HMO is not correctly identified as a Medicaid plan. On the other hand, she may over report Medicaid coverage, if her Medicaid coverage expires without her knowledge.
To probe for Medicaid coverage, surveys try different approaches. Surveys typically ask about Medicaid in the context of health insurance coverage, although a few ask about receipt of Medicaid benefits along with receipt of WIC, food stamps, and other government benefits/services. In Illinois, Medicaid coverage is tied to the tangible card itself, “Do you have a Medicaid card for yourself now?” The Georgia survey asks, “are you enrolled in Medicaid, Medicare, an employerprovided plan, or one you pay for on your own?” In Missouri, the interviewer lists “Medicaid or Medicaidpaid HMO” as one of the eight types of health insurance plans a person may have. There may some opportunities to analyze the effects of various wording differences as researchers compare answers to specific questions about Medicaid benefits with responses to the more general question about type of coverage and with data from administrative records.
Employer health plans. Eight surveys ask questions about employer health plans. These questions can be classified as falling into three categories:
I created a separate category for this third set of questions, because I was not sure whether “getting” health insurance benefits from an employer was more analogous to being offered health insurance, or actually enrolling in the employeroffered plan. I probably would have interpreted it as closer to “enrollment,” except that the Florida survey explicitly states to the interviewer that they “want to know if offered, not what employee is taking advantage of.”
Why not covered. Several instruments include additional questions to probe why a former recipient does not have health insurance coverage (four surveys); is not enrolled in Medicaid (three surveys), or is not participating in an employer health plan (two surveys).
Four grantees ask additional questions, not found on other state surveys. Illinois asks whether the leaver has applied for a Medicaid card. Missouri asks whether the children have been without medical coverage for any period since the family left cash assistance. If the family is covered by private insurance, Wisconsin asks who pays the premium. And Georgia asks a set of additional questions about child health coverage: When did you last have health insurance for the children? What was it? Has the child’s absent father paid for dental or medical expenses? For medical insurance?
The first two sets of survey items explored in this paper — questions about food stamp receipt and health insurance coverage — gather information about the family’s participation in programs other than TANF, and more generally, about work supports. Grantees also ask about many other types of program participation and work supports, most notably, child care assistance, but these are not included in the scope of this paper. The next two sections of the paper examine another important topical area covered by grantees, that of experiences of material hardship, deprivation, or insecurity. Most grantees ask questions about difficulty paying the rent or other bills, experiences with homelessness, use of emergency services, and the two subtopics examined below, food insecurity and lack of health care.
Food insecurity differs from the other four outcome measures in this review, because a “core module” of questions on food security/hunger was distributed to all grantees. This module was developed by a collaboration of Federal agencies and academic researchers, fielded in the April 1995 Current Population Survey (CPS) Supplement, and analyzed in a series of USDA reports released in September 1997.(10) Researchers from the U.S. Department of Agriculture encouraged the incorporation of this new set of questions in the leavers’ surveys, disseminating copies to all grantees and discussing it at the grantee meeting and through the listserve.
From the beginning, grantees were uncertain that they could incorporate the full set of 16 questions, four of which have multiple parts. USDA researchers, on the other hand, urged grantees to include all questions in their leavers’ surveys, because the full set of questions can be combined into a single overall measure of food security, drawing on 18 items from the 16 questions.(11) They were reluctant to promote a reduced form of the 18-item scale, arguing that the module was shorter than it seemed, since many questions are skipped if the household responds “no” to the initial screening questions. Inclusion of the full 18-item scale would also allow comparison to other surveys incorporating these items, such as the Survey on Program Dynamics (SPD). Grantees were concerned, however, that the questions seemed repetitive.
In the end, none of the grantees included the full core module with its 21 items (the 18-item scale plus a 3-item introductory screener). As shown in Table IV, the highest number of items included in a leavers’ study instrument was seven items, in the Cuyahoga/L.A. instrument. Three grantees — Arizona, South Carolina, and Wisconsin — ask only question (although they ask it about two time periods, while on welfare and since exit). The average grantee included 3.3 items. Although most (86 percent) of these items were drawn directly or with slight modifications from the USDA Food Security/Hunger Core Module, they were selected across the breadth of the survey instrument.
The most common question, asked by five grantees, is #8 in the USDA instrument (shown in Appendix IV-A), concerning whether adults in the household had cut the size of meals, or skipped meals because there was not enough money for food. Two grantees ask a follow-up question (#8a) about frequency of cutting the size of or skipping meals.
Another two grantees (Washington and Missouri) ask questions about meals being cut or skipped by children (#13, #14, #14a). Both previous research and analysis of the 18-item scale indicate that households tend to reduce food intake by adults before they reduce food intake by children and so reductions in meal sizes or meals by children is a greater sign of food insecurity and hunger. In fact, the items used by Washington and Missouri are used in the 18-item scale to distinguish households with “severe” hunger from households with “moderate” hunger.(12)
The questions about adults cutting the size of or skipping meals, #8 and #8a, are two of the six items included in a short (6-item) version of the USDA scale. This scale was included in the grantees’ resource books, but was not discussed much at the grantee meeting or the listserve, where the discussion focused on the full scale. This was perhaps unfortunate, because it turns out that the short scale is being used by four of the five states involved in the HHSfunded “Project on StateLevel Child Outcomes,” a project that adds child outcome measures to existing welfare reform evaluations.(13) Moreover, three of the six items in the short scale also are included in the Urban Institute’s National Survey of America’s Families (NSAF), which reports outcomes related to welfare reform in 13 states as well as on a national basis.(14) If grantees had known of its widespread use, they might have focused more attention on it. Instead, some dismissed it, because all six questions ask about adult rather than child experiences of reduced food intake and hunger, giving it the appearance of being illsuited for studies of families with children. In fact, the six questions about adult experiences with hunger can be used in households with children, because hunger in such households is likely to first show up as reduced food intake among adults, as discussed above. Another possible concern about the 6-item scale is that it cannot distinguish between households with “moderate” hunger and those with such “severe” hunger that children are affected. The latter group is relatively rare, however, and so may be difficult to identify accurately without use of the full 18-item scale.
Another common question, included in four leavers’ surveys, is #1, which asks about food insufficiency, that is, whether the household has enough food to eat and enough of the kinds wanted. A followup question probing for reasons for food insufficiency is asked in one survey (San Mateo). This question and its followups are not included in the short 6-item version, nor are they strictly part of the 18-item scale. The question is, however, is included in the Survey of Income and Program Participation, the Survey of Program Dynamics, and other national surveys, and has been used to measure food insufficiency over a 20-year period.
Two other common questions are #3, which asks whether the family found that the food that they bought just didn’t last, and #4, which asks whether the family found it could not afford to serve balanced meals. Four grantees, the Child Outcomes Project, and the NSAF are fielding #3, and three grantees and the Child Outcomes project are fielding #4. Both these questions are in the short (6-item) versions of the food security scale.
Other questions drawn from the Core Module were asked by one or two grantees, including:
Six items in the Core Module (#5, 6, 7, 11, 12a, and 15) were not asked by any grantee.
Finally, four grantees asked a question that was not directly from the USDA scale, namely, “was there a time when you had no way to buy food?” First used in the South Carolina survey, this question has been picked up by Wisconsin, Missouri, and Florida. If respondents answer yes, South Carolina and Wisconsin ask the respondent to state whether this occurred while the family was on welfare, or since exit. Florida also asks a question not included on other surveys concerning satisfaction with foods eaten by the household. The wording of these items is shown in Appendix IV-B.
Note that even when four or five grantees do ask the same general question about food insecurity, there may be variations which make it harder to make crossstate comparisons. The most important of these concerns time period. Although some grantees ask about food insecurity “during the past twelve months,” as in the Core Module, others ask over a 6-month time period, a 30-day time period, or the varying period since exit from TANF. Four grantees (Arizona, Illinois, South Carolina, and Wisconsin) ask the respondent to compare food insecurity in the 6-month or longer period since exit from welfare with food insecurity while on welfare. While this question gets directly at an issue of great interest to policy makers, it is not clear if respondents can accurately recall hunger experiences across two different 6-month time periods.
Only five food insecurity questions were asked by three or more grantees. These questions include three questions from the short version of the USDA scale (#8, #3, #4 ), one other question from the full core (#1), and a question not found on the USDA module (the South Carolina question). Four items were asked by two grantees, eight items by only grantee (including the Florida item not from the USDA module), and six items were not asked by any grantee. Overall, items on the short scale were asked by more grantees than other items in the full module: the 6 items on the short scale were asked by an average of 2.7 grantees, and the 15 other items from the Food Security/Hunger Module scale were asked by an average of 1.4 grantees.
In sum, comparisons of food insecurity across leavers’ studies will be limited by wording variations, and even more by the fact that grantees chose food insecurity items from across the broad array of 21 different items provided in the full USDA Food Security/Hunger Module. Opportunities will exist, however, for comparisons of food insecurity among leavers to food insecurity among more general national and state populations, given the inclusion of the short or full versions of the USDA module in such surveys as the SPD, NSAF, and welfare reform evaluations involved in the Project on StateLevel Child Outcomes.
Grantees did not ask as many questions about access to health care and health care status as they did about food security. Four grantees did not include any questions, four asked only question, and three asked three to five questions, as shown in Table V.
The most common question related to whether or not there was somebody in the home who was ever sick or hurt but unable to afford to get medical care. Six grantees asked a version of this question, which is asked in the National Health Interview Survey (NHIS), the Survey of Income and Program Participation (SIPP), and other national surveys. Five asked whether this had occurred “since leaving TANF.” In some cases this time interval may be four months, in other cases, nine to twelve months, depending on the timing of the survey interview relative to TANF exit in the various studies. The sixth grantee, the Cuyahoga/L.A. survey explicitly asks whether the adult or her children or spouse/partner have been unable to see a doctor or go to the hospital in the past twelve months. This has the advantage of being similar phrasing as in the NHIS and the SIPP, and so can be more easily compared. As in the SIPP, the Cuyahoga/L.A. survey repeats the question with regard to seeing a dentist. Two grantees (South Carolina and Wisconsin) ask about lack of access to needed health care during the time period the former recipient was on welfare, as well as since exit from welfare.
In Florida, the general question about not being to afford medical care when sick or hurt is supplemented by two later questions: whether the respondent needs medical care and cannot obtain it at the present time, and whether children are unable to obtain needed medical care at the present time. It will be interesting to learn how these “current time” questions fare.
Three grantees ask about health care status. Florida and Cuyahoga/L.A. use the same wording as in National Health Interview Survey and the Survey on Income and Program Participation, that is, “would you say that in general your health is excellent, very good, fair or poor?” Illinois asks how satisfied or dissatisfied the respondent is with her personal health and physical condition, and that of her children. Illinois also asks about satisfaction with the quality of the health care that the family can afford, and further, whether the respondent thinks, in terms of medical care for her children, she is better off, worse off, or about the same as when she left welfare. Finally, although not included in this analysis, a number of surveys include questions about personal or child health disability or sickness that may pose barriers to employment.
The final outcome measure included in this analysis concerns the former recipient’s knowledge of her potential eligibility for Medicaid and food stamps. Preliminary findings from administrative data and statefunded surveys of leavers suggest that relative low percentages of leavers — in some cases less than 50 percent — are receiving Medicaid or food stamps. This is a puzzling finding, given that many families who leave TANF are expected to be eligible for these programs, because of their relatively lowpaying jobs and low family income. Furthermore, all families leaving because of employment should qualify for transitional Medicaid assistance during the first six months after an exit. In order to better understand, and eventually, address issues of nonparticipation among this seemingly eligible population, many leavers’ surveys include questions about clients’ knowledge of their entitlement to benefits.
As shown in Table VI, seven instruments include questions that probe for respondents’ knowledge of their eligibility for Medicaid, their children’ eligibility for Medicaid, and their family’s eligibility for food stamps. Of the seven questions about Medicaid, five ask, “Did you know that... (you may be eligible, your children may be eligible, working adults are eligible, children are eligible)?” The Georgia instrument asks a somewhat broader question, “Once you leave TANF, do Medicaid benefits end?” The Illinois survey asks about the client’s perception of her specific eligibility, “Did you think you would be eligible to get a Medicaid card...?” In addition to asking about knowledge of eligibility rules, two surveys ask whether the former recipient received information from the welfare office about Medicaid.
Similar questions are asked about food stamp eligibility and information at welfare offices, although these questions are not as common as the questions about Medicaid eligibility. Though not included in this analysis, many surveys also include questions about recipients’ knowledge of eligibility for child care subsidies. (See Appendix V for a complete listing of the questions on knowledge of Medicaid and food stamps).
With regard to the questions about Medicaid eligibility, it seems to me that the question, “did you know that people like you are eligible for Medicaid...” is a “loaded” question, and thus not as meaningful as a more neutral question about eligibility rules, like the example in the Georgia survey. If someone were to ask me if I knew that the Federal government (my employer) provided a particular brand of health insurance or a particular kind of life insurance, I would likely answer yes, not wanting to admit that I do not recall the details of the ½" thick pile of papers on benefit options that I plowed through on my first day of employment. However, if one were to ask me more neutrally, does the Federal government offer X to its employees, I might be more likely to respond honestly that I do not know. Perhaps researchers might want to consider the potential merit of rewording questions about Medicaid eligibility, to bring them more in line with the more neutral wording of the Georgia example.(15)
One of the goals of this paper has been to investigate the extent to which the HHSfunded grantees developed common measures to track the economic and noneconomic wellbeing of families leaving welfare, and the extent to which they developed dissimilar items. Summing up what was found in each of the five outcomes:
In general, there was some degree of common measures, in the sense that many outcomes were measured across five or more instruments. For example, eight instruments ask about food stamp receipt in the current (or most recent) month, nine ask about types of health insurance coverage, five instruments ask if adults in the household cut the size of or skipped meals, six ask whether anyone in the household has been in need of medical care but unable to afford it, and five ask if the respondent knew she were eligible for Medicaid.
There are, however, differences in the wording of questions. Generally, these differences include variations in the time period measured (one month, six months, 12 months, the {X}months since TANF exit), whether the outcome was measured for the respondent, her children, her family or her whole household, and the context for the question (i.e., placement in the questionnaire, tone of the lead-in to the question, content of the immediately preceding questions).
In the area of food security, there was the possibility for incorporating identical, or at least very similar, questions across many different surveys, because all grantees were provided with a common module of questions. However, the overall length of the module — 16 questions with 21 items when counting multiple part questions — diminishes its usefulness as a common module for the leavers’ studies. Grantees are asking an average of only 3.3 items, and these items are drawn across a broad array of the 21 items. Of these 21 items, only 4 are found in 3 or more leavers’ surveys, 4 are found in two surveys, 7 are in just one survey, and 6 are not in any leavers’ survey. Grantees also are asking 2 questions not found in the USDA core module.
In hindsight, and with the benefits of this analysis, it is clear that more attention should have been focused on the short (6-item) version of the USDA food insecurity scale, the version used in the Project on StateLevel Child Outcomes and also, in part, in the National Survey of America’s Families. There would have been much more commonality in food security measures if grantees had been encouraged to draw from a pool of 6 items rather than a pool of 21 items.
In my role of coordinating technical assistance on the survey questions and other aspects of the leavers studies, I have sometimes felt caught in the middle, between grantees who want one or two questions to measure a certain outcomes, and experts in a particular area who propose a “core set” of 10 to 20 or more questions. The tensions experienced in the area of food insecurity were also evident in such outcomes as child care, child wellbeing, and other measures. On the one hand, I understood the arguments of content area experts, who, after careful analysis and sifting through of dozens of questions, produce a core set of questions that they do not think can be reduced further without compromising the validity of the measure. (In the case of food security, 58 questions were tested in a food security supplement and numerous factor analyses were conducted before the core module was boiled down to 16 questions and an 18-item scale). On the other hand, I understand the pressures on researchers conducting leavers studies to ask a minimal set of questions about each outcome measure. One lesson from this analysis, I believe, is that if the experts are unable to develop a condensed version of their outcome measures, each leaver study will develop its own condensed measure. At least that is what has happened in the area of food insecurity.
Although the 1998 round of HHSfunded leavers’ studies are mostly in the field at this point, the question of developing short measures of food insecurity or child wellbeing or other important outcomes is still quite relevant. States across the country are continuing to design and conduct studies of families leaving welfare. Moreover, my office is awarding additional grants to states and large counties within the next few weeks, to fund further studies of welfare outcomes. Although the second round will focus on families formally or informally diverted from welfare before entry, rather than the traditional “leavers,” the same questions are likely to be asked with regard to employment outcomes, program participation, and family and child wellbeing. I plan to use the results of this analysis in my work with the next round of HHSfunded grantees, as we work together to move closer toward the longterm goal of developing more common outcome measures to monitor the status of families affected by welfare reform.
1. The 14 grantees are: Arizona, District of Columbia, Florida, Georgia, Illinois, Massachusetts, Missouri, New York, South Carolina, Washington, Wisconsin, Cuyahoga County (OH), Los Angeles County (CA), and a consortium of San Mateo, Santa Clara, and Santa Cruz counties in California.
2. It was difficult to choose outcome measures for this analysis. The food security measure was chosen in order to examine how grantees responded to an 18-item food insecurity module that was developed by the U.S. Department of Agriculture (USDA) and disseminated to all grantees. The other outcome measures were selected partly as a complement to the food security measures and partly as a result of the high levels of attention being focused on issues of Medicaid and food stamp receipt.
3. Proposed sample sizes are much smaller — 300 to 330 — for three grantees (Cuyahoga County, Los Angeles County, and the District of Columbia). Two grantees, Florida and Georgia, proposed much larger sample sizes (15,000 and 7,800, respectively). Also note that one grantee — Missouri — proposed interviewing recipients a full 24 months after exit.
4. For discussion of the difficulty of making crossstate comparisons in the absence of common measures, see the General Accounting Office report, Welfare Reform: Information on Former Recipients’ Status, GAO/HEHS-99-48 (April 1999) and the Urban Institute issue brief, Where Are They Now? What States’ Studies of People Who Left Welfare Tell Us, a product of Assessing the New Federalism, Series A, No. A-32 (May 1999).
5. After extended discussion, all grantees came to agree on a common definition of leavers as cases that leave cash assistance and remain off cash assistance for a minimum of two months. Initially, some grantees had wanted to limit the study to those families that remained off welfare at the time of interview (as much as ten to twelve months after exit). This approach, however, severely limits our understanding of why some families are successful in remaining off welfare while others return to the rolls. Other grantees wanted to look at cases that closed for one month and then re-opened, because of the policy interest in understanding the effects of fullbenefit sanctions. Many “one-month” closings, however, are exits due to administrative churning, rather than “true” exits. The two-month definition was developed as a compromise.
Note that two states — Arizona and New York — plan to study “one month and longer” leavers, but will conduct analyses of the subgroup of “two-month” leavers in order to increase comparability across studies. Also note that while some grantees limit the study population to singleparent cases, some include two-parent cases and a smaller number include childonly cases. In most cases, however, all grantees are reporting at least some findings for the “singleparent two-month leavers” that are common across all studies.
6. More information about the guidance on administrative data outcomes or the electronic list serve for leavers researchers is available from the author at julia.isaacs@hhs.gov.
7. Further information about both the administrative and survey data in these topical areas is provided in Appendix A of ASPE’s Interim Status Report on Research on the Outcomes of Welfare Reform, posted on the ASPE/HSP web site at http://aspe.hhs.gov/hsp/hspwelfare.htm.
8. Cuyahoga and Los Angeles are two of the four sites in MDRC’s Study of Devolution and Change (Urban Change) and both sites are using the same instrument and similar analyses.
9. The Wisconsin survey explicitly defines immediate family, stating, “For the purposes of the next few questions, family members include only your children and your spouse, or the parent of at least one of your children.” The Cuyahoga/L.A. survey asks, “Did you receive any food stamps” and “Did anyone else in your immediate family receive food stamps?”
10. In particular, see Price, Hamilton, and Cook, Guide to Implementing the Core Food Security Module. Alexandria, VA: U. S. Department of Agriculture (September 1997).
11. During discussions of the food security questions at the first grantee meeting, there was some confusion between the 18-item scale and the questionnaire, which is numbered from #1 – #16. The 18-item scale is built on Questions #2 – #16 from the core module, including three follow-up questions (#8a, #12a, and #14a). The first question, a threepart question, is not used in the 18-item scale, but is included in the core module because of its in past surveys and its potential as a screener for the other questions.
12. The full 18-item scale has been used to classify U.S. households into four categories: food secure, food insecure without hunger, food insecure with moderate hunger, and food insecure with severe hunger.
13. Connecticut, Florida, Indiana, and Iowa are using the 6-item scale, as well as Question #1 from the 1997 version of the module. Minnesota, the fifth state involved in the operational phase of this project, is simply fielding Question #1.
14. The 13 states in the National Survey of America’s Families include Alabama, California, Colorado, Florida, Massachusetts, Michigan, Minnesota, Mississippi, New Jersey, New York, Texas, Washington, and Wisconsin. Researchers from Child Trends worked on the food insecurity and other child outcome measures in both the Child Outcomes Project and the NSAF.
15. An additional survey not included in this review, an Institute for Research on Poverty survey of welfare applicants in Milwaukee, asks families, “Can someone receive Medicaid (food stamps) without participating in W-2 [state name for TANF]?” This seems to me to be another example of a more neutrally phrased question.
[ HSP Home Page | ASPE Home Page | HHS Home Page ]
E-mail the author, Julia Isaacs, julia.isaacs@hhs.gov
Last modified November 5, 1999