Monitoring Outcomes for Former Welfare Recipients: A Review of 11 Survey Instruments. Conclusion

08/11/1999

One of the goals of this paper has been to investigate the extent to which the HHS­funded grantees developed common measures to track the economic and non­economic well­being of families leaving welfare, and the extent to which they developed dissimilar items.  Summing up what was found in each of the five outcomes:

  • Almost all (10 of 11) instruments ask questions about food stamp receipt, with these questions varying somewhat in terms of the time period of receipt and the definition of family or household receiving the benefit.
  • All surveys include questions about overall health insurance and Medicaid receipt (either through specific questions and/or through questions about type of coverage), for both the adult respondent and her children.  In addition, eight include questions about employer health plans, although these are not strictly comparable because of the potential difference between employers offering such coverage and employees actually enrolling in the plan.
  • Although all 11 instruments ask at least one question about food insecurity, these questions are drawn from the broad array of 21 different questions or follow-up questions included in the USDA Food Security/Hunger Core Module.  Only five specific questions are asked by three or more grantees.
  • Seven surveys ask about access to health care and/or health status, with six of these asking about whether the household could not afford to get needed medical care.
  • Seven instruments include questions about Medicaid eligibility and information, with five also asking about food stamp eligibility and information.

In general, there was some degree of common measures, in the sense that many outcomes were measured across five or more instruments.  For example, eight instruments ask about food stamp receipt in the current (or most recent) month, nine ask about types of health insurance coverage, five instruments ask if adults in the household cut the size of or skipped meals, six ask whether anyone in the household has been in need of medical care but unable to afford it, and five ask if the respondent knew she were eligible for Medicaid.

There are, however, differences in the wording of questions.  Generally, these differences include variations in the time period measured (one month, six months, 12 months, the {X}months since TANF exit), whether the outcome was measured for the respondent, her children, her family or her whole household, and the context for the question (i.e., placement in the questionnaire, tone of the lead-in to the question, content of the immediately preceding questions).

In the area of food security, there was the possibility for incorporating identical, or at least very similar, questions across many different surveys, because all grantees were provided with a common module of questions.  However, the overall length of the module — 16 questions with 21 items when counting multiple part questions — diminishes its usefulness as a common module for the leavers’ studies.  Grantees are asking an average of only 3.3 items, and these items are drawn across a broad array of the 21 items.  Of these 21 items, only 4 are found in 3 or more leavers’ surveys, 4 are found in two surveys, 7 are in just one survey, and 6 are not in any leavers’ survey.  Grantees also are asking 2 questions not found in the USDA core module.

In hindsight, and with the benefits of this analysis, it is clear that more attention should have been focused on the short (6-item) version of the USDA food insecurity scale, the version used in the Project on State­Level Child Outcomes and also, in part, in the National Survey of America’s Families.  There would have been much more commonality in food security measures if grantees had been encouraged to draw from a pool of 6 items rather than a pool of 21 items.

In my role of coordinating technical assistance on the survey questions and other aspects of the leavers studies, I have sometimes felt caught in the middle, between grantees who want one or two questions to measure a certain outcomes, and experts in a particular area who propose a “core set” of 10 to 20 or more questions.  The tensions experienced in the area of food insecurity were also evident in such outcomes as child care, child well­being, and other measures.  On the one hand, I understood the arguments of content area experts, who, after careful analysis and sifting through of dozens of questions, produce a core set of questions that they do not think can be reduced further without compromising the validity of the measure.  (In the case of food security, 58 questions were tested in a food security supplement and numerous factor analyses were conducted before the core module was boiled down to 16 questions and an 18-item scale).  On the other hand, I understand the pressures on researchers conducting leavers studies to ask a minimal set of questions about each outcome measure.  One lesson from this analysis, I believe, is that if the experts are unable to develop a condensed version of their outcome measures, each leaver study will develop its own condensed measure.  At least that is what has happened in the area of food insecurity.

Although the 1998 round of HHS­funded leavers’ studies are mostly in the field at this point, the question of developing short measures of food insecurity or child well­being or other important outcomes is still quite relevant.  States across the country are continuing to design and conduct studies of families leaving welfare.  Moreover, my office is awarding additional grants to states and large counties within the next few weeks, to fund further studies of welfare outcomes.  Although the second round will focus on families formally or informally diverted from welfare before entry, rather than the traditional “leavers,” the same questions are likely to be asked with regard to employment outcomes, program participation, and family and child well­being.  I plan to use the results of this analysis in my work with the next round of HHS­funded grantees, as we work together to move closer toward the long­term goal of developing more common outcome measures to monitor the status of families affected by welfare reform.