Minutes of the Technical Assistance Workshop, May 3-5, 2000. Topic 1: Opportunities and Difficulties in Large Urban Survey Research


Speaking at this session was Kristin Shook of the Northwestern University Institute for Policy Research. Shook is the manager of the Illinois Family Study. Shook began by introducing herself and naming some of her collaborators on the Illinois Family Study, including Principal Investigator Dan Lewis of Northwestern, Stephanie Rieger of the University of Illinois at Chicago, Paul Klepner of Northern Illinois University, and James Lewis of Roosevelt University.

The Illinois Family Study (IFS) is a six-year longitudinal study of approximately 1,500 families that were receiving Temporary Assistance to Needy Families (TANF) in late 1998. Most had at least one child younger than 18 years. The IFS looks at a variety of different outcomes, primarily related to the grantees. In most cases, the grantee is the mother of the family. The issues and the purposes of the IFS are to explore employment trajectories of women receiving welfare as they transition off of welfare. The IFS, in part because of Shook's interests, also includes a number of questions that focus on children. Shook noted that the legislation that mandates the ISF says that it is not to be funded by the Illinois Department of Human Services. It does receive some funding from other state agencies and from local and national foundations. The possibility of federal funding exists.

Shook said that "researchers and state policymakers often come at the questions from a slightly different angle" but that the IFS was a unique opportunity because the state Department of Human Services is very interested in understanding what about welfare reform is working and what is not. IFS has an advisory panel on this study of key state legislators that gives direction about the kind of information that would be very useful for them in future years and the project has a good working relationship with DHS, allowing them access to the kind of data systems they need. Other agencies and advocates are also very involved.

Earlier Surveys

Shook noted that the IFS team has also relied on existing surveys related to welfare research. In particular, she mentioned:

  • The Three Cities study--a panel study of welfare recipients--done by P. Lindsey Chase-Lansdale of Northwestern
  • The Women's Employment study at University of Michigan by Shedon Danziger and a large group there
  • The New Hope Project also conducted at Northwestern by Greg Duncan

The measures and survey tools used by these studies helped the IFS team develop their measures, allowing for comparisons and for some certainty that the materials and measures had been tested.

Administrative Data

Shook also noted the that they were able to link their survey data to existing administrative data from Illinois human service agencies and that they had worked with Chapin Hall to develop their ability to use and understand the administrative data.

Child Well-Being Indicators

Shook described how she, working with Northwestern's Duncan, had included indicators of child well-being into the survey. She and Northwestern pediatrician Dana Hall found support from the National Institutes of Health (NIH) for a supplemental study on child neglect. This supplemental study will involve 500 families from the original sample who had children younger than three years at the time of sampling.

Shook was asked, "Is your expectation that there will be a high level of various kinds of neglect?"

She replied that work already underway in Illinois using administrative data had examined the links between child welfare and cash assistance systems prior to TANF. Some of that earlier work had shown "that about 5 percent of a cohort of AFDC recipients prior to TANF became involved in the child welfare system over the course of the year. And that was any form of involvement, an indicated report, a case opening with child protective services." Based on these and other estimates, she expects:

. . . approximately 75 families a year that had some involvement with the system. And over time the cumulative involvement gets as high as 25 percent over a five-year period. So with the 1500 families we did some power analyses to make sure to get the results we are looking for, and also with the 500 families as well.

They are also interested in changes in Medicaid coverage--even temporary losses of coverage--and changes in health care and medical neglect rates. She continued:

The advantage of looking at these links within the context of welfare reform is that there is variation produced in each of these independent variables I described that is automatically being produced by welfare reform policy. So we are taking advantage of that variation to explore these links to different forms of child maltreatment. And given that in Illinois about almost two thirds of all the kids who enter foster care in a given month come from a family that recently received welfare, exploring this within group variation was also important to us. So often you see poverty or welfare used as kind of a predictor for child protection intervention, but nobody really understands what that means.

The research will also explore what subgroups of the population are having difficult times.

Shook then moved to a secondary goal of the supplemental child well-being study, to carefully define what they mean by child neglect and different forms and child maltreatment. She said that there are three sets of outcomes that the researchers are interested in:

The first kind of precursor set of outcomes has to do with severe risk. Situations we can identify that are likely to present a potential harm to children. So this is beyond being a single-parent family or something like that. It's the presence of multiple hazards in the home, being without health-care coverage for a significant period of time.

There is a second stage that looks at harms to kids--accidents, injuries, poisonings. The things we link back to medical records, information from children's pediatricians, and health-care practitioners that they see to try to get a better understanding of whether this is a pattern or is this something that just happened once. We want to make sure that we're kind of separating out things that happened normally and something that is likely to be a sign of more problematic family functioning.

And a third level is looking at how these different sets of outcomes relate to child protection intervention. So often in child welfare research the outcome used as an indicator of child abuse or neglect is involvement with the child protection system.

When asked how long the supplemental study will take place, Shook replied that the NIH funding is for five years, but that they plan to:

make sure we get enough baseline information that it will be convincing to funders later on. We plan that to go all out during the first year at least to make sure we really get a good set of baseline indicators. Given that we have all this access to these other systems, in recognizing that this is a really unique opportunity, I have to be strategic about convincing people about this. It's interesting, in groups like this people understand that. But a lot of times people say, for example when we go to foundations, "Oh, this is a study and deals with welfare, we've funded enough of these welfare studies at this point." So it's really been more of a struggle than I thought it was going to be. I thought it was quite clear about how unique this opportunity was, but apparently not to everyone. But we're getting there, it's just going to take a little more time before it is not our main focus all the time.

Earlier in her talk, Shook had said that one of the project's goals was obtaining a careful definition of neglect. At this point, she elaborated on that, saying:

. . . there are some real issues in identifying neglect. When we say we are identifying neglect, we are asking questions in a retrospective way. We are relying on combinations of indicators so that in the context of the interview you're not identifying neglect based on the answers to one of the questions or an obvious combination of the questions. We are not just relying on combinations of indicators for that purpose, we set this up theoretically in terms of the way you can go about identifying treatment and ways you can't. But you need to use combinations of questions not just one question to really see what's going on. And we're also relying on the medical records very heavily, which are based on past interactions with health-care professionals. And there are lots of conflicting feelings about whether we need to err on the side of figuring out that something is going wrong in the family and then reporting. Or realizing that so little is understood about the etiology of neglect we have to keep that in mind as we're tracking the families over time, and not set this study it up in a way that kind of subverts its own purpose right from the get go. I don't know if this is really the right time to get into a discussion of ethical issues. I'd be glad to. I haven't quite decided where I fall on the issue. So the physical abuse, a lot of the questions we're asking would be about punishment strategies. Things based on the common tactics scale. But we're not asking a lot of the more severe items because we learned in pilot testing that these were very insulting to a lot of families. I myself did not feel comfortable asking a lot of these really severe questions about punishment techniques. So we probably have enough in there to begin to identify some kind of a threshold but not get at the extreme end of the continuum.

An audience member asked: If you have that administrative data you would be able to track allegations or people who are at least classified by the system. Shook suggested that:

Some of this might come up in medical chart review too. We might detect some suspicions that physicians have about abuse. But the neglect indicators are less problematic because, for instance, the environmental neglect indicators are about food insufficiency using the U.S. Department of Agriculture's Food Insufficiency Scales. We gear them really towards children's experiences, more so than adults' experiences, of food insufficiency. And we have different levels of risk just like the U.S. Department of Agriculture does. We have mild, moderate and severe food insufficiency. So we try to kind of set up our indicators in a way that there's more of a continuum and we are not just kind of looking for some threshold yes/no but exploring this in a little more of a complex way.

Respondent Permission to Access Medical Records and Other Administrative Records

An audience member asked how the IFS team got permission to use medical records. Shook explained that respondents signed two consent forms, one giving permission to do the study and the second granting permission to access information from "a long list of administrative data systems, school records, medical charts," and other datasets. This avoided going back to families later on. Shook reported that about 85 percent of respondents gave their consent to access the data during the first wave and they're hoping to boost that during subsequent waves as trust grows. (Later, Shook said that she thought state entities did not need permission to examine administrative data in this way, but as a private consortium of researchers, the IFS team felt like they did need respondent permission.)

An audience member suggested that families connected with protective services might be reluctant to grant permission to access records, creating bias. Shook explained that they could assess such a bias because they drew their sample from administrative data and can draw a similar sample for comparison.

An audience member asked if the study collects data on household composition in order to examine the relationship it might have with good or bad outcomes for children. Shook said that they were getting at this by using household rosters and asking about the relationships among everyone living in the household.

In response to a question, Shook said respondents are paid $30 for each annual interview.

Data Collection


The IFS sampling frame included addresses, telephone numbers, and information from DHS records back five or six years. All this information, plus the increasing use of the Link card (an electronic debit card for benefit transfer), eased locating tasks. Also, addresses for families receiving childcare subsidies tended to be more accurate than were some other records. Despite this, locating difficulties have slowed the first wave of the study. In response to a question, Shook said that those respondents who are found to have moved out of state will be interviewed by telephone using a shortened instrument.

Scheduling & Refusals

Shook described the challenge of scheduling interviews and the care the staff take to treat respondents nicely. Regarding hard versus soft refusals, she said:

We find that probably seven times out of ten people will say no on the initial phone call and when you callback even just the next day, "Oh, yeah, come on out." So you have to be good at learning the distinctions.

In-Person Versus Telephone Survey

Shook described two limitations of collecting data over the phone. First was the need to use a shorter instrument. Second is that respondents often lack consistent telephone service.

Interviewer Safety

A participant asked about safety for the interviewers. Shook said that she and her team had relatively little difficulty and she discussed some of her experiences collecting data in high-crime areas. In response to a related comment about the difficulties one survey had using school teachers as interviewers, Shook recommended using social work students.


For the IFS, locating poses a greater challenge than refusals. They have a high response rate among people they actually reach. Among the strategies they have adopted for finding people are canvassing neighborhoods on foot. They also looked at additional databases. In response to a question, she did say it was her sense that those who were really difficult to find were more likely to be not receiving assistance. There was discussion then about who might be missed both in this study, and in other studies of human service recipients in times of transition.

Qualitative Interviews

Shook closed by noting that the IFS plans to conduct qualitative interviews with a 10 percent subsample of families who have multiple involvements with the human services system to see how they experience those involvements.