(May 4, 2000)
[ Main Page of Report | Contents of Report ]
Speaking at this session was Bong Joo Lee of Chapin Hall. Each participant in the session introduced her- or himself and explained why she or he was attending. As a prelude, Bong defined a data warehouse, saying that it is an integrated information system that is primarily used for decision support in contrast to a client-tracking system. Client-tracking systems need real-time capability, a data warehouse is not and cannot be made to yield real-time data. A data warehouse links data from a variety of sources to form a coherent picture.
Administrative Data
Lee pointed out two advantages offered by administrative data over survey data. One was that they provide information on a complete population of interest rather than a sample. This will allow economical analyses of very small subpopulations. Expense can be further reduced because administrative data are collected as an integral part of the functioning of government departments and agencies. If you can find a way to use administrative data, then you don't need to incur the additional expense of collecting new data.
Record Linking
Linking records involves
Deterministic linking methods give equal weight to different types of information a record may contain. For example, a deterministic approach might place equal reliance on a match between the names on two records or a match between two birth dates.
Probabilistic approaches allow the researcher to exploit the probability that a match on particular items is more or less likely to indicate that the individuals named on two records are, in fact, the same individual. For example, birth date information is subject to errors made by a mistake on a single digit, and the number of possible birth dates is relatively small. Names, in contrast, are more likely to be recognizable even if a single error is made. Probabilistic linking allows the researcher to create an approach that weights the value of matches on such information appropriately.
Robert Goerge of Chapin Hall led the session. Goerge asked the audience about their areas of interest in order to tailor his examples. Four of the audience members had health backgrounds. The others represented education, child welfare, and human services in general. A data warehouse was defined as a repository for storing integrated information for efficient querying and analysis. A data mine was defined as a more specific data repository designed for the analysis of particular questions. Bob indicated that the many states were making efforts to put data in one place through a data warehouse and indicated that the focus of the session would be on warehousing individual-level data.
A member of the audience asked for clarification on the definition of a data mine. She thought that a data mine was more targeted than a data warehouse. Bob indicated that it was not only the specificity of the subject of the data mine that distinguished it from a data warehouse, but also the way in which the data are connected. Goerge diagramed a data warehouse. In the example, he depicted a warehouse that contained a master index of all individuals and tables on Medicaid recipients, TANF recipients, foster children, children in special education, individuals receiving childcare subsidies, and immunization data. Goerge pointed out that a common identifier linked each table. He suggested that each table or each topic area could actually be thought of as an individual data warehouse in addition to the entire set functioning as a data warehouse and, by way of example, showed a diagram of a database that Chapin Hall created for the Illinois Department of Children and Family Services (DCFS).
The DCFS Database combines two areas in the child welfare system: abuse and neglect reporting and child welfare services. Goerge indicated that this data warehouse links the two areas by identifying common individuals using probabilistic record linkage. He emphasized that only when one is certain that individual data are linked correctly could one analyze the data. He added that conducting analysis from a full data warehouse is cumbersome and that what is done is to create smaller datasets that contain summary information. These smaller datasets are data mines.
One of the audience members raised the question of confidentiality concerns across agencies. They were also interested in whether or not confidentiality was a concern within agencies. Goerge pointed out that one of the best ways to relieve confidentiality concerns was to emphasize the fact that the data will only be presented in an aggregate format, say in groups of 5 or more individuals, and will not be used for case management. He said that he was not aware of any confidentiality breeches and that because of concerns for confidentiality, some information, such as birthdates, might be excluded from datasets. He also described security procedures to restrict access to data.
Goerge offered examples of agencies that have little trouble acquiring outside data. Child support was the best example of a agency that has been granted access to almost all data by the legislature. Children in foster care are Medicaid eligible so child welfare agencies are granted access to their Medicaid claims data. Child welfare and TANF agencies regularly exchange data in order to determine IV-A funding eligibility.
Updating. To be useful, warehoused data must be unduplicated and up to date. Illinois updates its data warehouse monthly. The decision of how often to update is critical to the effort it takes to maintain the data warehouse.
Geographic identifiers. An audience member asked if the address is used to link records. Goerge said not usually used unless one knows that the address is from the current time period. He added, however, that having some piece of geographic data--such as a county code--improves the quality of the match. To do more small area data analysis, data are geocoded.
Standardization. Standardization is required across databases and how to standardize service events is a major issue. The standardization process involves reconciling the master index and the events across the data warehouse and resolving all inconsistencies before analysis.
Efficiency. Efficient information storage requires information to be stored once in the data warehouse. A second key to efficiency is linking events. For example, within a master index, we want to know then relationship between individuals. A link table that presents the relationship between pairs of individuals would significantly boost efficiency. Linking individuals or events and storing that data in link files is a good way to make queries more efficient.
Summary
A data warehouse is a good way to deliver information to the users and to store information safely. It provides end users with a single data model and a query language. Goerge said that Florida and Ohio had two of the largest data warehousing efforts that he knew of. He pointed out that, in response to welfare reform, many states were interested in linking their TANF and child welfare system data in order to measure the effect of welfare reform on the child welfare system. Many data warehouses are designed specifically for this research. Regarding standardization of database formats, the audience and Goerge pointed out that there is no national center for social program statistics unlike the areas of health and education. It was thought that a national center would benefit the standardization of data definitions and the development of data warehouses.
Fred Wulczyn and John Dilts of Chapin Hall led the Small Area Analysis session.
Geographic Information Systems and Small Area Analysis
Introduction
In his introduction, Wulczyn announced that Asher Ben-Arieh would lead a discussion around issues he raised in his talk the day before. Wulczyn then provided a cursory history of small area analysis, tracing its origin in Chicago to the early days of the Cook County juvenile court a century ago, when the court began to map the addresses of the children coming before the juvenile court. The recognition that many came from the same neighborhoods helped foster an investigation of the ways in which neighborhoods were implicated in child development and that neighborhood flaws, not the character flaws of children and families, might result in children coming to the court's attention. Early geographic mapping was done by hand. A purpose of this session is to cover the process of conducting small area analysis and to cover some of the terminology associated with it. It is also meant to show how maps can be both useful tools and can create pitfalls. Wulczyn said of maps, "As is so often the case, pictures do say 1000 words, but there are at least 1000 words that are unsaid by those pictures and we want to show you both sides of that scenario in our presentation today."
Geographic Information Systems
Dilts set out to discuss geocoding and mapping. He defined geocoding and sketched his purposes, saying:
Geocoding is the process of taking data of any sort and making it analyzable in terms of its spatial characteristics. That sort of analysis can be a visual analysis, the kind you do when you're drawing maps, but it can also be statistical analysis. There are all kinds of techniques that statisticians have used and geographers have used for a long time to analyze data that has geographic characteristics attached to it.
There are two aspects to geocoding. One is the standardization of address records. The other is the process of matching that standardized data to its geographical reference point.
Standardization is necessary because address records are particularly complex and are typically stored as strings of data. Such strings can be difficult for computers matching to interpret unless they are standardized. Steps in standardizing can include separating the elements of the address into units that that can be recognized for matching, like house numbers, street names, or zip codes; by cleaning out extraneous information; and by always using the same abbreviation to represent the same thing. The approach to standardization depends upon the number and quality of records. A few hundred records might be best standardized by hand. But large record volumes or data of varying quality might be best standardized by computer. Chapin Hall uses a standardization program called AutoStand.
Dilts discussed how addresses, once standardized, could be matched with a particular geographic location by the use of geographic reference files, such as the Census Bureau's TIGER (1) files or other files. TIGER files contain geographic coordinates and Census information for every address within a given area. There are two methods of matching the address to a location--deterministic matching and probabilistic matching. Such desktop packages as Map Info, Arc info, or Map Point perform deterministic matching in linking addresses and geographic records. Data problems can interfere with the reliability of deterministic matching, making probabilistic matching a better choice. Probabilistic matching is done by a specialized software. Chapin Hall uses Auto Match. Auto Match links records on the basis of information that is similar, within certain parameters that are set by the user. It's called probabilistic because the matching is based on an underlying statistical probability that two records are in fact the same record. Once you've done this you can move on to the fun step of trying to analyze your data either visually or statistically.
Questions
Wulczyn asked for questions.
An audience member asked if Auto Match users can set their own levels of probability? Dilts replied:
Yes, absolutely. The software allows total control of the match in terms of the amount of importance that you want to attach to various elements. It might be very important that the house number match exactly, but not so important that street name match exactly. You may allow a little bit of difference in the spelling of the street names that sort of thing. It allows you to determine those levels. It also allows you to determine the overall levels of certainty that you want that the two match.
A number of state representatives described the mapping software in place in their areas and on the difficulties of obtaining the necessary specificity. Wulczyn commented:
On the geocoding side, the lowest level of granularity is the address. Literally, the TIGER file is latitude and longitude coordinate for every address that the census has. And from that point you can wrap it in any kind of political or social subdivision, zooming in and out depending upon the situation. The issue, if you only have zip codes you don't necessarily have to go through a geocoding process, you already have a piece of geographic information. The question is how well does that overlap with political, social, or other types of subdivisions.
Mapping
Like statistical analysis, mapping helps tease out the patterns in the data and make them apparent. With a map, information can by layered by overlaying successive slides containing different types of related information. Wulczyn and Dilts showed a dot-density map of the homes of substantiated abuse and neglect cases. That information was layered on a map of Chicago that contained the boundaries of the city's 77 community areas. Of that map, Dilts said:
You can begin to impose a little order on the data and allow comparisons to be made. But dot-density maps are themselves pretty limited because they don't allow you to . . . quantify comparisons between the areas. You can eyeball this and draw the conclusion that some areas have much less than others, because there's a density clustering around these dots, but that's about all you can say. Given that, the thing that one has to do next is to begin aggregating the data within these areas and to create what's called a thematic map.
Wulczyn added, "The question here is . . . how do you use the maps to tell an effective story? That's essentially one of the things that we're trying to do within the social indicators context is to be able to describe what's happening." Wulczyn then showed how examining smaller geographic areas and by studying the density of incidence of abuse and neglect rates, the user can adjust the way he or she constructs a mental model "about the underlying social dynamics in these areas." He further developed this illustration by showing maps of small geographic areas that included both abuse and neglect rates and rates of AFDC (Aid to Families with Dependent Children) receipt for small areas. He showed how the seeming relationship between high rates of abuse and neglect among families receiving AFDC indicated by the map was an oversimplification.
The presenters continued adding information to the map, explaining how far mapping would take them and noting the point when they reached the "limits to how much insight these visually appealing maps actually impart." By combining the maps with line charts, the presenters matched rates of child abuse and neglect cases with rates of AFDC participation to illustrate the relative levels of incidence, a comparison that helped reveal a weak relationship between high rates of AFDC participation and high rates of child abuse and neglect reporting. Wulczyn said
So what was visually appealing upon closer examination, again the story shifts a bit. It suggests there is something more subtle going on. We have spikes here. The point is that in adjacent areas, adjacent not in a geographic sense, not neighboring Census tracts in a geographical sense, but neighboring areas with respect to how many AFDC participants they have or the rate per thousand, in that sense of a neighbor, they could have very different child abuse and neglect reporting rates. And maps are as likely to obscure that point as they are to reveal the point. So (we must) go back to a more standard X/Y plot... Remember, we started out this series of slides with the traditional X/Y plot . . . . we're not abandoning the need to look at things in a more traditional way because it reveals that there may be problems with imparting too much causality to the relationship between AFDC and child abuse and neglect reporting rates.
A member of the audience said,
If you were looking at Child Protective Services (CPS) and wanted to do an intervention in that area, what this is saying to me is that maybe the common thinking about the relationship between CPS and income isn't holding true. And then you would want to map some other things like foster care, family size, ethnicity, and that would start to tell you more and you would start to change your variables.
Wulczyn replied:
That's right. And that's the advantage, you're essentially creating multiple links to a variety of standardized data sources. And so the Census data will give you some things about household structures, so on and so forth. If you have other sources of administrative data or survey data that you can integrate with this sort of larger schemata of information, it maintains the organization of information so that the process of going through and generating questions isn't delayed because you have to get the information together. It allows you to do things in a very interactive way and change your perspective in a real-time sort of way whereas in the old days it would take a long time to get the information.
During the remainder of the session, there was some discussion of mapping software, including costs and limitations. Ann Segal said that Jake Jacobson of Charlotte, North Carolina was using a variety of mapped data, including foster children's residence, Medicare receipt, bus routes, utility shut-offs, and other information in order to help planning for multi-generational daycare centers that would serve as senior centers and child care centers. In addition to mapping, he evaluated children and used other tools. Segal noted that Jacobson's maps are not public documents, but are used for internal planning purposes, a way of protecting confidentiality.
Speaking at this session were
Mairéad Reidy
Reidy began the session by recapitulating some of the previous discussions on school readiness indicators, including some themes of the December 1999 meetings on school readiness measures held in Providence. These included the ambiguity of the term "school readiness" and the need for consensus definitions of what children should be ready for and what schools need to do to be ready for children. She showed a series of slides:
Hosted by Rhode Island KIDS COUNT
Providence, December 2 & 3, 1999
This meeting was sponsored by the Advancing States' Child Indicators Initiatives project of the Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services, in partnership with the Carnegie Corporation of New York Starting Points Initiative.
Meeting Objective
To assist states and communities participating in the ASPE Child Indicators Project and the Carnegie Starting Points Initiative to develop practical indicators that can be used:
School Readiness
What Children Know and Can Do
Physical Well-Being and Motor Development
Social and Emotional Development
Child and Family Connections
Services Provided
Ready Schools
Data Collection
Speaker John Love said that:
Communications Strategies
Love recommended that we:
Data Interpretation
At the New England meetings, Love suggested states:
Childcare
Childcare Measures that Could be Examined Using Existing Data
Quality
Affordability
Accessibility
Elizabeth Burke Bryant
Bryant said that the state of Rhode Island is very serious about developing school readiness indicators because of the major investments the state has made in childcare during the past two years. Legislative leaders will continue to support the expansion and improvement of Rhode Island's early care and education system if they have evidence that these investments are having a positive impact in terms of children arriving at school ready to learn. Because time was of the essence, the Rhode Island Child Indicators team decided that there is no perfect set of measures, and we moved ahead in selecting a logical set of indicators that are a good reflection of a young child's health, cognitive ability, and social competency. Some of these indicators were already being tracked by Rhode Island Kids Count; others had to be developed and put in the field. Rhode Island added childcare quality questions to our market rate survey of childcare providers (now in the field), and we are adding school-readiness questions to our survey of kindergarten teachers. A complete listing of Rhode Island's school readiness indicators was distributed as a handout for the workshop participants.
Reeva Sullivan Murphy
Murphy said that her background as an early childhood educator and kindergarten teacher served her well in her new position as state Childcare Administrator. As part of her work with Rhode Island's Child Indicators Team, she has been developing indicators that, taken together, paint the full picture of a child's readiness for school, and the school's readiness for each child. It is the combination of good health, and a comfort level in interacting with other children that make some children more ready than others in a kindergarten teacher's view. Murphy said that using the market rate survey every two years will be a great way to obtain data on school readiness, and that the willingness of the people who administer the comprehensive School Accountability for Learning and Teaching (SALT) (2) survey to every public school student will provide the kindergarten data that we need. She encouraged other states to select a list of indicators that cover the key areas and that will make sense to policy makers when it comes time to report on whether the state's early childhood investments are making a difference.
David Murphey
David Murphey of Vermont picked up on the idea of consensus in the area of school readiness. He said that over the past year, a degree of consensus has emerged among those working on school readiness indicators and discussed some of the different approaches he sees states taking in understanding school readiness. Murphey saw consensus around these ideas:
Murphey identified these as different paths states are taking:
A
B
Murphey discussed Vermont's approach to assessing school readiness.
Overhead (3) A Chronology of Vermont's Strategy: A 3-Part Assessment of School Readiness
Vermont has used focus groups of interested parties to receive input on this plan and to obtain information on ways to revise their instruments. This fall, Vermont plans to use this assessment statewide. This survey will replace an earlier survey that the state had found less useful than it hopes the new assessment will be. Murphey made draft instruments available to the audience. In response to questions from the audience, Murphey said that they plan on asking teachers questions on their own experiences and history. He also described the anticipated ways in which the Vermont data collection would be conducted.
Before Beatrice Colón began, Martha Moorehouse said that Rhode Island's initiatives in the area of school readiness are motivated in part because they are making big investments in this area. She asked if the work in Illinois and Vermont was taking place in the context of changing investments and the desire to track those investments. Murphey said that there is strong support in Vermont for standards-based assessment but that it remains to be seen if there will be a funding investment.
Beatrice Colón
Beatrice Colón said that about six years ago, the State Board of Education decided that teachers needed a systematic way to collect and assess data on what children should know and what they do indeed know. A small group of teachers and administrators developed a system to use work sampling as an alternative to testing. The system has three components: developmental checklists, portfolios, and summary reports done three times a year. These reports provide a chance for the teacher to reflect on what the child has gained. They can be shared with the parents, too. Colón pointed out that Illinois has 450 prekindergarten programs. Not every one of those programs is using work sampling. Each district can choose what kind of assessment they will use. At most levels, the work sampling system is being implemented using only one to two domains as a start. A few systems are using all of the domains.
The developmental checklist follows the domains. Those domains are:
Some of the systems that tried to implement work sampling using all domains had a difficult time doing so.
Colón said that an integral part of the system is continuous training and staff development. It is her view that continuous staff development must be built in to the process and school administration must support it. Without those steps, the system will fail. Because there are 450 prekindergarten programs (excluding Chicago) and more than 400 classroom programs in Chicago alone, Illinois State Board of Education (ISBE) contracts out some of this staff development work. A challenge to work sampling is the variation in the ways in which principals implement it and support, or fail to support, staff development.
As a result of the work sampling, and of questions and suggestions from the school district, ISBE has developed and is further developing indicators, outcomes, and standards for early childhood programs. Currently, Colón has a draft document that is being reviewed. They are also developing program standards for birth-to-three programs.
Colón said that Illinois does not collect data, per se, on work sampling. But ISBE does collect data on children in the prekindergarten program. There is a record for each child and family that contains health, safety, education, and other data. There is also a longitudinal study that follows a sample of children through 12th grade, but that the population of interest is very transient.
David Ayer
David Ayer said that, in the mid-1990s, Maryland developed its Maryland Model for School Readiness. They wanted a way of assessing children by the time they reach the end of their kindergarten year. The work sampling system, which covers a great deal of ground, was chosen to do that. Since 1997, there has been a pilot implementation underway. All of the twenty-four school systems in Maryland will use this in kindergarten in the coming school year or the one following. They will look at all of the domain areas available in the work sampling system. Maryland will examine the aggregate percentages of children at the end of kindergarten reaching a proficient level. This information will be incorporated into Maryland's indicators.
Monica Herk
Monica Herk identified four Georgia benchmarks. These are:
Herk focused mostly on the GKAP, created by legislative mandate to assess the readiness of kindergarten students to enter first grade. Herk provided eight pages of background information on the Georgia Benchmarks and on GKAP. Summarizing the GKAP findings over the past four years, she said that the numbers have been relatively unchanged. (A handout put this number between 87.3 and 88 percent over the period.) She said that the Georgia example shows that, although the state has made a huge investment in prekindergarten education, the measure doesn't show a change. However, evaluation studies that have tracked children are showing that children who go through the state-funded prekindergarten program seem to score better on the kindergarten assessment.
With the most recent revision of the GKAP (GKAP-R) some of the earlier data will not be comparable to later data. One of the changes made in this revision aligned the GKAP-R more closely with the Georgia Quality Core Curriculum (4) content standards. This alignment will also mean that Georgia's scores will not be comparable to data from other states. There have also been changes in methods of administration in an effort to reach a form of administration best suited to children of this age.
The GKAP-R has three windows of assessment, one at the beginning, one at the middle, and one at the end of the school year. This supports the use of the GKAP-R as a diagnostic tool for the teacher.
Janel Harris
Janel Harris said that the February 2000 meeting on promotional indicators held in St. Paul had tried to address three goals:
Harris felt that they made progress on all of these goals, but that the promotional indicators effort remains very much a work in progress. One conclusion they drew was that they needed to further examine the research that underlies some of the areas in which indicators are needed so that they could see what research relates to those things that seem intuitively to have value as promotional indicators. Harris also commented on two talks on indicators by federal agency staff--Martha Moorehouse of ASPE and Casey Hannon of the CDC. Harris felt that most of this federal work was deficit-oriented and she found very little in the way of positive or promotional indicators. She said that, none the less, she felt it was important to push forward the promotional indicators agenda, in part because people are "really tired of being viewed as a collection of problems." Promotional indicators can provide a more balanced picture of the situation being examined.
Steve Heasley
Steve Heasley went over what has happened with promotional indicators since the Minnesota meetings. He said that it is important to point out that a lot of the interest in promotional indicators stems from frustration with traditional indicators that are deficit oriented. He said that it is interesting that frameworks are developed "to measure, 'child well-being,' but, because of the limitations of the data, what we are measuring is the absence of well-being." The philosophy before promotional indicators is to provide measures that do capture well-being--those that are related to strengths and assets.
Heasley said that he was struck by Ben-Arieh's presentation and the work on the indicators worldwide.
Some of the areas in which promotional indicators could be developed include:
Heasley said that the cross-state email group is open to anyone who wants to participate, not just to the four states that convened the February meeting.
{Go To Contents]
Speaking at this session was Kristin Shook of the Northwestern University Institute for Policy Research. Shook is the manager of the Illinois Family Study. Shook began by introducing herself and naming some of her collaborators on the Illinois Family Study, including Principal Investigator Dan Lewis of Northwestern, Stephanie Rieger of the University of Illinois at Chicago, Paul Klepner of Northern Illinois University, and James Lewis of Roosevelt University.
The Illinois Family Study (IFS) is a six-year longitudinal study of approximately 1,500 families that were receiving Temporary Assistance to Needy Families (TANF) in late 1998. Most had at least one child younger than 18 years. The IFS looks at a variety of different outcomes, primarily related to the grantees. In most cases, the grantee is the mother of the family. The issues and the purposes of the IFS are to explore employment trajectories of women receiving welfare as they transition off of welfare. The IFS, in part because of Shook's interests, also includes a number of questions that focus on children. Shook noted that the legislation that mandates the ISF says that it is not to be funded by the Illinois Department of Human Services. It does receive some funding from other state agencies and from local and national foundations. The possibility of federal funding exists.
Shook said that "researchers and state policymakers often come at the questions from a slightly different angle" but that the IFS was a unique opportunity because the state Department of Human Services is very interested in understanding what about welfare reform is working and what is not. IFS has an advisory panel on this study of key state legislators that gives direction about the kind of information that would be very useful for them in future years and the project has a good working relationship with DHS, allowing them access to the kind of data systems they need. Other agencies and advocates are also very involved.
Earlier Surveys
Shook noted that the IFS team has also relied on existing surveys related to welfare research. In particular, she mentioned:
The measures and survey tools used by these studies helped the IFS team develop their measures, allowing for comparisons and for some certainty that the materials and measures had been tested.
Administrative Data
Shook also noted the that they were able to link their survey data to existing administrative data from Illinois human service agencies and that they had worked with Chapin Hall to develop their ability to use and understand the administrative data.
Child Well-Being Indicators
Shook described how she, working with Northwestern's Duncan, had included indicators of child well-being into the survey. She and Northwestern pediatrician Dana Hall found support from the National Institutes of Health (NIH) for a supplemental study on child neglect. This supplemental study will involve 500 families from the original sample who had children younger than three years at the time of sampling.
Shook was asked, "Is your expectation that there will be a high level of various kinds of neglect?"
She replied that work already underway in Illinois using administrative data had examined the links between child welfare and cash assistance systems prior to TANF. Some of that earlier work had shown "that about 5 percent of a cohort of AFDC recipients prior to TANF became involved in the child welfare system over the course of the year. And that was any form of involvement, an indicated report, a case opening with child protective services." Based on these and other estimates, she expects:
. . . approximately 75 families a year that had some involvement with the system. And over time the cumulative involvement gets as high as 25 percent over a five-year period. So with the 1500 families we did some power analyses to make sure to get the results we are looking for, and also with the 500 families as well.
They are also interested in changes in Medicaid coverage--even temporary losses of coverage--and changes in health care and medical neglect rates. She continued:
The advantage of looking at these links within the context of welfare reform is that there is variation produced in each of these independent variables I described that is automatically being produced by welfare reform policy. So we are taking advantage of that variation to explore these links to different forms of child maltreatment. And given that in Illinois about almost two thirds of all the kids who enter foster care in a given month come from a family that recently received welfare, exploring this within group variation was also important to us. So often you see poverty or welfare used as kind of a predictor for child protection intervention, but nobody really understands what that means.
The research will also explore what subgroups of the population are having difficult times.
Shook then moved to a secondary goal of the supplemental child well-being study, to carefully define what they mean by child neglect and different forms and child maltreatment. She said that there are three sets of outcomes that the researchers are interested in:
The first kind of precursor set of outcomes has to do with severe risk. Situations we can identify that are likely to present a potential harm to children. So this is beyond being a single-parent family or something like that. It's the presence of multiple hazards in the home, being without health-care coverage for a significant period of time.
There is a second stage that looks at harms to kids--accidents, injuries, poisonings. The things we link back to medical records, information from children's pediatricians, and health-care practitioners that they see to try to get a better understanding of whether this is a pattern or is this something that just happened once. We want to make sure that we're kind of separating out things that happened normally and something that is likely to be a sign of more problematic family functioning.
And a third level is looking at how these different sets of outcomes relate to child protection intervention. So often in child welfare research the outcome used as an indicator of child abuse or neglect is involvement with the child protection system.
When asked how long the supplemental study will take place, Shook replied that the NIH funding is for five years, but that they plan to:
make sure we get enough baseline information that it will be convincing to funders later on. We plan that to go all out during the first year at least to make sure we really get a good set of baseline indicators. Given that we have all this access to these other systems, in recognizing that this is a really unique opportunity, I have to be strategic about convincing people about this. It's interesting, in groups like this people understand that. But a lot of times people say, for example when we go to foundations, "Oh, this is a study and deals with welfare, we've funded enough of these welfare studies at this point." So it's really been more of a struggle than I thought it was going to be. I thought it was quite clear about how unique this opportunity was, but apparently not to everyone. But we're getting there, it's just going to take a little more time before it is not our main focus all the time.
Earlier in her talk, Shook had said that one of the project's goals was obtaining a careful definition of neglect. At this point, she elaborated on that, saying:
. . . there are some real issues in identifying neglect. When we say we are identifying neglect, we are asking questions in a retrospective way. We are relying on combinations of indicators so that in the context of the interview you're not identifying neglect based on the answers to one of the questions or an obvious combination of the questions. We are not just relying on combinations of indicators for that purpose, we set this up theoretically in terms of the way you can go about identifying treatment and ways you can't. But you need to use combinations of questions not just one question to really see what's going on. And we're also relying on the medical records very heavily, which are based on past interactions with health-care professionals. And there are lots of conflicting feelings about whether we need to err on the side of figuring out that something is going wrong in the family and then reporting. Or realizing that so little is understood about the etiology of neglect we have to keep that in mind as we're tracking the families over time, and not set this study it up in a way that kind of subverts its own purpose right from the get go. I don't know if this is really the right time to get into a discussion of ethical issues. I'd be glad to. I haven't quite decided where I fall on the issue. So the physical abuse, a lot of the questions we're asking would be about punishment strategies. Things based on the common tactics scale. But we're not asking a lot of the more severe items because we learned in pilot testing that these were very insulting to a lot of families. I myself did not feel comfortable asking a lot of these really severe questions about punishment techniques. So we probably have enough in there to begin to identify some kind of a threshold but not get at the extreme end of the continuum.
An audience member asked: If you have that administrative data you would be able to track allegations or people who are at least classified by the system. Shook suggested that:
Some of this might come up in medical chart review too. We might detect some suspicions that physicians have about abuse. But the neglect indicators are less problematic because, for instance, the environmental neglect indicators are about food insufficiency using the U.S. Department of Agriculture's Food Insufficiency Scales. We gear them really towards children's experiences, more so than adults' experiences, of food insufficiency. And we have different levels of risk just like the U.S. Department of Agriculture does. We have mild, moderate and severe food insufficiency. So we try to kind of set up our indicators in a way that there's more of a continuum and we are not just kind of looking for some threshold yes/no but exploring this in a little more of a complex way.
Respondent Permission to Access Medical Records and Other Administrative Records
An audience member asked how the IFS team got permission to use medical records. Shook explained that respondents signed two consent forms, one giving permission to do the study and the second granting permission to access information from "a long list of administrative data systems, school records, medical charts," and other datasets. This avoided going back to families later on. Shook reported that about 85 percent of respondents gave their consent to access the data during the first wave and they're hoping to boost that during subsequent waves as trust grows. (Later, Shook said that she thought state entities did not need permission to examine administrative data in this way, but as a private consortium of researchers, the IFS team felt like they did need respondent permission.)
An audience member suggested that families connected with protective services might be reluctant to grant permission to access records, creating bias. Shook explained that they could assess such a bias because they drew their sample from administrative data and can draw a similar sample for comparison.
An audience member asked if the study collects data on household composition in order to examine the relationship it might have with good or bad outcomes for children. Shook said that they were getting at this by using household rosters and asking about the relationships among everyone living in the household.
In response to a question, Shook said respondents are paid $30 for each annual interview.
Data Collection
Locating
The IFS sampling frame included addresses, telephone numbers, and information from DHS records back five or six years. All this information, plus the increasing use of the Link card (an electronic debit card for benefit transfer), eased locating tasks. Also, addresses for families receiving childcare subsidies tended to be more accurate than were some other records. Despite this, locating difficulties have slowed the first wave of the study. In response to a question, Shook said that those respondents who are found to have moved out of state will be interviewed by telephone using a shortened instrument.
Scheduling & Refusals
Shook described the challenge of scheduling interviews and the care the staff take to treat respondents nicely. Regarding hard versus soft refusals, she said:
We find that probably seven times out of ten people will say no on the initial phone call and when you callback even just the next day, "Oh, yeah, come on out." So you have to be good at learning the distinctions.
In-Person Versus Telephone Survey
Shook described two limitations of collecting data over the phone. First was the need to use a shorter instrument. Second is that respondents often lack consistent telephone service.
Interviewer Safety
A participant asked about safety for the interviewers. Shook said that she and her team had relatively little difficulty and she discussed some of her experiences collecting data in high-crime areas. In response to a related comment about the difficulties one survey had using school teachers as interviewers, Shook recommended using social work students.
Nonresponse
For the IFS, locating poses a greater challenge than refusals. They have a high response rate among people they actually reach. Among the strategies they have adopted for finding people are canvassing neighborhoods on foot. They also looked at additional databases. In response to a question, she did say it was her sense that those who were really difficult to find were more likely to be not receiving assistance. There was discussion then about who might be missed both in this study, and in other studies of human service recipients in times of transition.
Qualitative Interviews
Shook closed by noting that the IFS plans to conduct qualitative interviews with a 10 percent subsample of families who have multiple involvements with the human services system to see how they experience those involvements.
The Data Standardization session was led by Allen Harden of Chapin Hall. Harden said that the session might be better named data "decomposition and standardization," because you can't standardize data without thinking about how to break it apart, and noted that the aim of the session is to show how to standardize data for purposes of comparison.
Introduction
Measurement. Using a variety of illustrations, Harden addressed the need to devise measures suited to the topic of interest. When thinking about constructing measures, it is important to recognize early on what you are measuring, that is, what is the unit of analysis. In this work, it is often a child, but might be other things, such as attributes of families or places. Sometimes a statistic, an absolute magnitude, is all a manager cares about. But, this is rare. It is more common for the policy community to need more information, such as information for comparison. Comparisons can require rates, means, unduplication, and other adjustments of data.
When making comparisons among data, it is important to understand the definition of the terms. Harden used kinship care as an example, noting that different state definitions of kinship care inhibited comparisons of foster care population statistics. He pointed out the importance of interviewing the sources of the data to ensure that you know what generated the measure, how it is defined, and who is included in the counted group.
Population of reference. Along with the measure, it is necessary to define the population of reference, the denominator of a proportional measure. A simple percentage is the share of children with particular attributes. That prevalence statistic, may, by itself, be sufficient.
But trying to get information on what produced the statistic in the first place requires a more complicated approach. This is especially true if the numerator is an event or something else that is countable, then it can be important to define the denominator as the population that can be appropriately be thought to be at risk of experiencing this event. When we can define an event and the likelihood of that event occurring to a population of interest, this is an incidence statistic. (Harden noted that because the language used to define these measures comes out of public health and epidemiology, it is negative sounding, including phrases like "at risk." He noted that one can be "at risk" of success.)
Using the example of child welfare, Harden said that the initial prevalence statistic can represent two things--rate of entry (or incidence) and also duration in care.
Comparisons. Harden moved through an array of comparisons to which measures might be put. These included over time, by age of population, by region of residence, by gender, and others. He pointed to the work done by demographers on fertility as an example. A crude measure is the birthrates for a particular populations, more informative are age-specific birthrates that get at the timing of births.
An audience member said that the idea of standardization was attractive, but much of the data he encountered was compromised. He asked for guidance on how much compromise was too much. Harden said that these determinations are made based on what the data user ultimately wants to know, when he or she needs to go out and get better information. He also noted the importance of attaching the right explanations to the data so that there is no confusion about what the data can and cannot explain. Harden showed some comparisons of child populations in Chicago.
Discussion
Discussion revolved around appropriate uses of data.
Leading this session were Robert Goerge and Bong Joo Lee of Chapin Hall. It covered:
Point-in-Time Versus Longitudinal Analyses
One way to look at foster care is through point-in-time analyses. Another way is to look at the history of an entire entry cohort. Each approach gives different information. The latter approach will reveal time spent in care and show the distribution of care over time. The longitudinal approach is useful in examining what effects the length of time spent in care and point-in-time analysis can overrepresent the number of children who spend a long time in care.
Ecological Fallacy
Using aggregated data to make inferences at smaller geographic levels can be complex. Bong uses the example of infant mortality rates across geographic areas. By examining the relationship among key risk factors and infant mortality rates, Bong can account for about 1/3 of the seeming geographic variability in those rates. By considering such factors as the utilization of prenatal care, more of this variability can be explained.
Small Cell Size
The small size of populations in some geographic areas means that random events can produce substantial variation in incidence, service use, or other statistical measures applied to those areas. One way to compensate for small cell sizes is to simply analyze larger populations. Another possibility is to combine cohorts across years to create a multiyear moving average that compensates for the possibility of extreme variation.
Adjusting Rates with Regression
Lee also discussed the use of regression analysis to help untangle the influences on different variables in a community. In the discussion that followed, Lee and the participants covered how regression analyses can be used in planning and explaining what data show to communities.
Because of the widespread interest in the international indicators work of Asher Ben-Arieh, a special session was scheduled in which those interested could ask him questions. Ben-Arieh began by noting that the researchers working on this international effort--about 25 in all--are involved in this in addition to their regular work. He also said that the international working group did not pay anyone's travel or lodging to these five-day meetings.
Ben-Arieh said that the northern European nations are progressive on many children's issues and that New Zealand offers very innovative programs and research. He noted, in particular, New Zealand's antipathy toward achievement and, in agreement with a member of the audience, noted that the New Zealanders like to measure systems rather than individuals. Ben-Arieh also singled out the Australians for their innovative research on the cost of children.
In response to a question about why Ben-Arieh preferred the terms "positive" and "negative" indicators to "promotional" indicators, Ben-Arieh said that the idea of "promotional" indicators, like the idea of outcomes, characterized children as part of a process.
1. Produced by the U.S. Bureau of the Census, TIGER (topically integrated geographic encoding and referencing) files are a digital database of geographical features and political boundaries used to create maps. For more information, see the website of the Census Bureau, www.census.gov.
2. On its website, the Rhode Island Department of Elementary and Secondary Education calls its School Accountability for Learning and Teaching (SALT) program "a school-centered cycle of activities to improve school and student performance in the Rhode Island public schools." An annual survey of students, parents, and teachers helps inform the development of school improvement plans. For more information, see http://www.ridoe.net/default.htm.
3. This overhead is reproduced from notes taken by a staff member and may differ from Murphey's original.
4. Georgia's 1986 Quality Basic Education Act mandated the development of a uniform core curriculum called the Quality Core Curriculum. The Quality Core Curriculum must be included in the basic curriculum provided to students in Georgia public school districts. For more information, see the Georgia Department of Education's Georgia Learning Connections homepage, www.glc.k12.ga.us/qstd-int/homepg.htm.
Main Page of Report | Contents of Report
Home Pages:
Advancing States' Child Indicators Initiatives
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)