Minutes of the Technical Assistance Workshop, May 3-5, 2000

This technical assistance workshop was the third in a series of technical assistance workshops hosted by the Chapin Hall Center for Children for participating states in the Child Indicators Initiative. The Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services, sponsors the Advancing States Child Indicator Initiatives Project. Martha Moorehouse is the Project Officer. The Chapin Hall Center for Children at the University of Chicago provides technical assistance to the Indicator Project states and prepared these minutes. Harold Richman of Chapin Hall is the Principal Investigator and Mairéad Reidy is the Project Director.

Chapin Hall Center for Children
1313 East 60^th Street
Chicago, Illinois 60637
773/753-5900
A Chapin Hall Working Paper, CS-67

Overview: The technical assistance workshop, held at Chapin Hall on May 3-5^th 2000, was the third in a series of technical assistance workshops hosted by the Chapin Hall Center for Children for participating states in the Child Indicators Initiative. The workshops encouraged peer leadership and collaboration among states, and provided states with an opportunity to work with and learn from one another on areas of common interest. Chapin Hall discussed technical assistance needs extensively with participating states and worked collaboratively with them to develop the agenda for this meeting. State participants reported on progress, shared information on successes and brainstormed around emerging and ongoing challenges. Key experts in the field of child indicators were invited to share their expertise and brainstorm with participants as they discussed challenges and successes. The May 2000 workshops provided participants with practical guidance in areas including: (1) how to move from a more general list of conceptualized indicators to a set of operationalized indicators that can be measured and tracked; and (2) how to use these operationalized indicators in policy making at the state and local levels. Key themes included the following:

International Perspective: Asher Ben-Arieh, Associate Director of the National Council for the Child in Israel, and Editor of the State of the Child in Israel, directs an international project entitled "Monitoring and Measuring Child Well Being." Asher described the work of this project and offered an international perspective on the child indicator work.

From Concept to Indicator: Allen Harden, Chapin Hall Center for Children, led a discussion on how to appraise progress, identify critical remaining issues, and decide where to focus next efforts. Discussion included a review of the general topic of indicator development, including guidelines, strategies, and limits for indicators of child well being.

Data Management: Topics covered included:

1. Data Linkage - Bong Joo Lee, Chapin Hall Center for Children
2. Data Mining - Robert Goerge, Chapin Hall Center for Children
3. Small Area Analysis - Fred Wulczyn and John Dilts, Chapin Hall Center for Children
4. Urban Survey - Kristin Shook, Institute for Policy Research, Northwestern University
5. Data Standardization - Allen Harden, Chapin Hall Center for Children
6. Interpretation of Data - Robert Goerge and Bong Joo Lee, Chapin Hall Center for Children

Revisiting School Readiness and Promotional Indicators This session was led by Mairéad Reidy, Senior Research Associate at Chapin Hall; Elizabeth Burke Bryant, Executive Director of Rhode Island KIDS COUNT; Reeva Sullivan Murphy, Childcare Administrator in the Rhode Island Department of Human Services; David Murphey, Senior Policy Analyst in the Vermont Agency of Human Services; Beatrice Colón of the Illinois State Board of Education; David Ayer, Director of Research, Evaluation and MIS in the Maryland Governor's Office for Children, Youth, and Families; Monica Herk, Project Coordinator of the Georgia Child Indicators Project; Janel Harris, Research Scientist in Minnesota's Department of Health, Division of Family Health/MCH; and Steve Heasley, Consultant to the West Virginia Governor's Cabinet on Children and Families.The panel discussed how states were measuring components of school readiness and how states are using school readiness indicators to form and monitor policy.

From Indicators to Outcomes: Considerations and Strategies for Communities: This session was led by Ada Skyles, Chapin Hall Center for Children; Arlene Andrews, Institute for Families in Society, University of South Carolina; Michael Bennett, Egan Urban Center, DePaul University; Jennifer Jewiss, University of Vermont; David Murphey, Vermont Agency of Human Services; Mary Nelson, Bethel New Life, Inc.

Community-based organizations and other community actors are prime audiences for and potential users of the information represented by indicators. This session briefly recapped discussions from earlier meetings on making indicator data accessible and useful to communities, and then moved on to the next level of development: How can we help communities move from indicators to outcomes? That is, how can indicator data be made a real force in community action? This session explored the range of effective and appropriate roles for states in support of this goal, both in working with communities to understand the reality behind the data and in helping communities interpret data and employ it for their own purposes. The session informed discussion from a number of perspectives, including state government, community-based organizations, and the research community, and covered a range of focuses for action (rural, urban, etc.)--all with the intention of further informing discussion and action at the community level.

Institutionalizing and Sustaining the Use of Indicators: This session was lead by Fred Wulczyn, Chapin Hall Center for Children; David Ayer, Maryland Governor's Office for Children, Youth and Families; Thomas Darling, Schaefer Center for Public Policy, University of Baltimore; Christine Johnson, Learning Systems Institute, Florida State University; Jim Witherspoon, Maryland Department of Human Resources. This session focused on how to extend the use of indicators both internal and external to government. State experiences were presented that described how agencies can use indicators to sharpen state policy and program development within government and in government's relationships with communities.

"

Opening Session

Opening Remarks

Harold Richman, Director of Chapin Hall, opened the meeting by welcoming the representatives of the Indicators Project states and sketched the plans for the meeting. Ann Segal of the Office of the Assistant Secretary for Planning and Evaluation welcomed those attending. Martha Moorehouse reviewed the history of the Indicators Project meetings, acknowledged Minnesota and Rhode Island for hosting earlier meetings, and expressed excitement about what the states had accomplished so far. State delegation leaders introduced delegation members and mentioned particular interests.

An International Perspective

Mairéad Reidy introduced Asher Ben-Arieh, Director of the Center for Research and Public Education at the National Council for the Child in Israel. His presentation was anchored by a series of overheads. These follow.

Monitoring and Measuring Children's Well-Being
New Trends and Domains

The International Project

Who We Are

• More than 70 experts from 27 countries from a variety of disciplines (social work, law, sociology, pediatrics, psychology, economics, statistics, and more).
• Coming from NGOs, government institutions, universities, foundations and international organizations.

The International Project Goals

To reexamine "old" indicators of children's well being and compose a new set that will:

• Use the child as the unit of observation
• Go along with the concepts of children's rights and childhood as a stage of itself
• Be based on a variety of sources of information
• Include positive indicators
• Be policy oriented

To suggest and invent ways and methods to use this field for promoting the well-being of children.

To prepare and impose a work plan for a multinational cluster of studies on the well-being of children.

The International Project Outcomes

All papers presented in our first meeting in Jerusalem (25 papers) were published in a special Eurosocial Report (No. 62). This volume presents the rationale and basis of our project and the thoughts that guided us through our work. A number of academic papers and a book were written and are either published already or are due to be published shortly.

The International Project Outcomes

A list of guidelines suggesting how to compose and use indicators in order to promote children's well-being has been agreed upon. The group has decided to avoid the minefield of suggesting a single theoretical framework for children's well-being.

The International Project Outcomes

We decided to work with a modified set of five domains of children's well-being with which we had become familiar. The domains are: economic resources and contribution, personal life skills, civil life skills, safety and health status, and children's activities

A list of some 50 indicators for all five domains was composed. All indicators adhered to the principles and guidelines of the project and all are theoretically based. A description on how they could be measured or whether there are any existing data sources accompanies each suggested indicator.

A Study of the State of the Child Reports

This study was based both on an extensive literature review, which examined seven electronic databases and collected data from an international network of informers regarding "state of the child" reports around the world. The databases used were: the U.S. Library of Congress, University of California library network, University of Newcastle library, University of Toronto library, Sydney University, University of South Africa and University of Hong Kong. The database research has resulted in 2,561 titles. After a thorough title screening, we were left with 241 titles and, after a thorough content screen, with 101. The international informers added 151 titles. After eliminating the duplicates, we had 134 state of the child reports to analyze and study.

Building a Common Language

One of the first things that struck us was the absence of any commonly widespread terms or language in the field. It seems therefore that any effort to analyze and learn from published reports, would have to start by clearing the fog that accompanies the discussion in the field. We therefore suggest the following terms and concepts as a basis for any future discussion.

• The target population. The target population is the population of children to which the report refers. We suggest the following two-option dichotomy. The whole child population meaning reports that cover all the children within a given geographical area. A segment of the child population means reports that cover only part of the child population identified by special need, special situation, age group, ethnic origin, or religion.
• The target audience at whom the report is aimed at. We identify several options. The general public and the media, policy makers, professionals working with children, advocates, the academic world, and children themselves.
• The report orientation. We suggest two possibilities. Service oriented, meaning the report focused on the availability of services for children and the consequences of specific programs, services, or policies. Children oriented, meaning a focus on the status of children without necessarily connecting them with specific services and without looking directly on the availability of services for children.
• The unit of observation. This phrase refers to the focus of the data collection. We know that in most data collection efforts the unit of observation was the family. We even know that if we would like to have better knowledge of children's lives, we need to focus on the child and not his household or family. Another distinction is between the child as a unit of observation rather than the program or the service.
• Domains are subtopics within the various reports, either as a separate chapter or a section devoted to a specific issue. We suggest a two-option dichotomy. Traditional domains, when the subtopics are divided by the different social services or professions. "New" domains, when the subtopics are defined as interdisciplinary and cut across services and focus on the child.
• Survival and beyond. We argue that measures that deal mainly with death and life, as well as with the basic needs of children, should be referred to as survival indicators. Any other indicators that measure the state and quality of life of children should be known as beyond survival indicators.
• Positive and negative indicators. We suggest the following language, negative indicators are indicators that are measuring the existence of harmful aspects in children's lives or their absence (i.e. child abuse or injuries). Positive indicators are indicators that measure the existence of desired and positive aspects of children lives (i.e. success in school, supporting family and leading an healthy life style).
• Types of reports. Reports that consist of a number of chapters, each devoted to a different domain, should be known as multi-topic reports. Those that are focusing only on one aspect or domain of children status are single-topic reports. Integrated-topic reports are those that use an integrated index for children's well-being.

Where is the Field Going?

Based on our international project and the state of the child report study, we are confident that the field is going through 4 major shifts:

• From survival to beyond
• From negative to positive
• From well becoming to well-being
• From "traditional" to "new" domains

What are the Consequences of Those Shifts?

We need to redefine the concept of children's well-being

We need to redefine the domains of children's well-being

We need new indicators

We need more data

Our Domains of Children's Well-Being -- Civil Life Skills

Concentrating on children's civic skills is vital for their well-being. Both in regard to its immediate meaning in children's lives and for their participation in long-term community, national, and global political life. We need to learn to what extent children are acquiring the knowledge, skills, values, and attitudes that are fundamental to democracy. We have decided to divide this domain into 3 subdomains: civic/community values and awareness, civic/community activities, and opportunities for civic/community activities.

Our Indicators-Civil Life Skills

Civic and community values and awareness:

• Percentage who report an interest in current events and in social problems
• Degree of support in tolerance and expression of minority views and other forms of civil rights
• Perceived importance of contributing to the community and society

Civic and community activities:

• Percentage of children who belong to and are active in an organization (political, community service, religious, or general)
• Percentage of children who report having political, religious, or social discussions with family and friends
• Percentage of children who volunteer

Opportunities for civic and community activities:

• Percentage of children who are attending schools with student governments
• Degree of child participation in decision making about their lives
• Adult/government reaction to children's participation
• Opportunities for voluntary work of children
• Belief in one's ability to bring about change

Our Domains of Children's Well-Being--Personal Life Skills

We found it helpful to divide this domain into 3 subdomains: interpersonal skills and resources, academic skills and resources, intrapersonal skills and resources. This gives a sense of going from social interactions with others, through developing skills for learning and then to skills of dealing with one's self.

Our Indicators-Personal Life Skills

Interpersonal skills and resources:

• Support from family, friends, and others
• Conflict resolution skills
• Communication skills
• Behavior among and with peers group

Academic skills and resources:

• Literacy and numeracy level
• Technological knowledge level
• Level of "general knowledge" (i.e. history, arts, geography, culture)

Intrapersonal skills and resources:

• Anxiety, depression, and general well-being
• School behavior
• Perceived well-being
• Perceived self-efficacy

Our Domains of Children's Well Being--Safety and Physical Status

Safety and physical status are commonly thought of as the most basic components of well-being. A child who is not safe will likely neither live nor develop in an optimal way and may also be more vulnerable to physical injury and trauma. We have decided to split it into two sub-domains, i.e., safety and physical status. We believe such a differentiation will clarify the content of the domain. However, it may be that physical status is a subset of safety.

Our Indicators-Safety and Physical Status

Safety indicators:

• Prevalence of the use of corporal punishment of children
• Prevalence of child abuse and neglect in its various forms.
• Perception of safety among children
• Exposure of children to environmental hazards
• Rate of hospitalization due to trauma
• Rate of child death by age and cause

Physical status indicators:

• Substance (tobacco, alcohol and drug) abuse
• Height, weight, and body mass index measures
• Level and incidence of physical activity
• Eating habits and diet

Our Domains of Children's Well-Being--Economic and Social Resources

We argue that children are not merely an economic burden on society (or the family). Children are themselves an economic resource and furthermore, children are active actors and contributors within their household or the broader economy. We have divided the domain into 4 subdomains: macroeconomics and distributive justice, expenditures on children, access to resources, and children's contribution and autonomy.

Our Indicators-Economic and Social Resources

Macroeconomic and distributive justice:

• Relative child poverty rates before and after taxation and transfers
• Percentage of public expenditures by age groups

Expenditures on children:

• Average costs of children (for the household and for society) by age group
• Percentage of family expenditure contributed (spent by or on) children

Access to resources:

• Measures of children share of the family material and economic resources
• Access to various social, educational, and health services regardless of economic status

Children's contribution and autonomy:

• Benefits/transfers paid directly to children or to family on behalf of children
• Sources of income of children
• Percentage of family resources contributed by children
• Children's perception of their contribution to the family resources

Our Domains of Children's Well-Being--Children's Activities

Across political jurisdictions and cultures, children engage in work, play, creativity, consumption, social interactions, and other activities that are analogous to adult activities yet qualitatively different. Children are active in their families, among peers and community groups and in various social institutions such as schools, informal education, recreation, and information networks. Indicators in this domain may relate to the extent of engagement in activities, the nature of the activities, places in which these activities take place, and the children's perceptions of the relative importance and contribution of the different activities

Our Indicators-Children Activities

Distribution of children's time across types of activities

• Percentage of time spent in productive activities (school, paid work, household work, and contributing to community)

Percentage of time spent in obligatory vs. voluntary activities

• Distribution of children's time across different participants (with family, alone, with other children, with other adults, or with other children and adults)
• Percentage of time spent in places not designated specifically for children
• Distribution of children's time by satisfaction levels

Lessons Learned

• Children's well-being is culture contingent, no culture could impose its beliefs or norms on others
• Children are not a resource, they are human beings
• The task is enormous thus there is room for everyone and almost any effort, the only condition is that it is aimed at improving child well-being
• This is an emerging and dynamic field which has gained more and more interest around the world
• To build a true international partnership, you have to have a flat hierarchy organization
• The fact that no one owned the project contributed to its success
• Having project members come as individuals and not representatives of their institutions, enabled us to concentrate on what we saw as important and interesting and not to be obligated to any particular view or issue

Prospects For the Future

We hope to achieve the following goals:

• Have better knowledge on children's well-being and lives
• Enhance public awareness of the state of children and the need to promote their well-being
• Use the indicators and the knowledge derived from them as a tool for advocacy
• Use the indicators for evaluating programs, services, and policies for children
• Put children in the focus of social policy
• Contribute, at least a bit, to the ongoing effort to improve children's well-being

Questions

In response to a question about cultural differences in how countries plan to use data, Ben-Arieh suggested reading a paper by a University of Hong Kong faculty member on the importance of discipline.

Janel Harris asked if the activities of children might not be easily defined as positive or negative, an assertion with which Ben-Arieh agreed. She also asked if surveys were reliable ways to collect data on children's activities. Ben-Arieh said that he was convinced that survey data were reliable on children's activities down to a respondent age of six years. In response to other questions, Ben-Arieh said that he was not satisfied with what had been learned so far about children, and would not be satisfied until data were collected from children themselves about their lives. He also called middle childhood a neglected area of study.

Session 1: From Concept to Indicator (May 3, 2000)

The aim of Chapin Hall's Allen Harden was to present a broad overview of indicators and to take a middle ground between the use of indicators and the development of results-based strategies. He began by offering definitions of indicators and of child well-being. Regarding child well-being, he said that everyone has a general idea about child well-being, but that the topic is complex, multidimensional, culturally dependent, and subjective. In sum, he called child well-being the union of objective conditions and subjective values. Harden moved on to indicators. He identified a number of roles that have been sketched for indicators, including those listed below.

• Describing conditions of children
• Monitoring change
• Setting goals
• Structuring program accountability and assisting in evaluation

He noted that some of these roles can involve tasks for which indicators are not well-suited, such as identifying causality, and made particular reference to the difficulty of being sure that an indicator is measuring what you think it should measure. As an example, Harden said that a variety of policy-related and other factors come into play in measuring school readiness and noted that, among the factors that help drive interest in measuring school readiness is the fact that we can measure it. That is, school entry provides the first opportunity to touch data on an entire cohort of children. After running over these ideas, Harden used the example of school readiness to illustrate the intersection of factors and concerns regarding measurement. He noted three orientations toward school readiness:

• Children need to be ready for school so that they can participate;
• School readiness is an outcome measure of social investment in childcare and other early childhood supports; and,
• School readiness is itself an early childhood outcome.

These approaches provide similar information about children, but the motivations for seeking this information, and the uses to which those who desire the information expect to put it, can be different. Those motivations and anticipated uses can become a source of conflict among stakeholders, even if those stakeholders have broad goals for children's well-being held in common. Assessment of individual children serves as one example of how uses of data can be the subject of dispute. Harden carefully sketched some of the factors that contribute to school readiness and some of the underlying disagreements about them or the prejudices that audiences bring to them. He moved from there to measures and possible sources of data to support those measures and the difficulties in accessing some data sources.

In the course of his presentation, Harden employed a number of overheads. Three of these follow. One presents a conceptual model of data relations. The second details those relationships as they relate to school readiness. The third looks at ways of observing children.

Slide 1:

Slide 2:

Slide 3:

Session 2: Special Topics, (May 4, 2000)

Part 1

Topic 1: Data Linkage

Speaking at this session was Bong Joo Lee of Chapin Hall. Each participant in the session introduced her- or himself and explained why she or he was attending. As a prelude, Bong defined a data warehouse, saying that it is an integrated information system that is primarily used for decision support in contrast to a client-tracking system. Client-tracking systems need real-time capability, a data warehouse is not and cannot be made to yield real-time data. A data warehouse links data from a variety of sources to form a coherent picture.

Administrative Data

Lee pointed out two advantages offered by administrative data over survey data. One was that they provide information on a complete population of interest rather than a sample. This will allow economical analyses of very small subpopulations. Expense can be further reduced because administrative data are collected as an integral part of the functioning of government departments and agencies. If you can find a way to use administrative data, then you don't need to incur the additional expense of collecting new data.

Record Linking

Linking records involves

• Standardizing and cleaning data to ensure compatibility and avoid duplicate records
• Selecting a deterministic or probabilistic linking method.

Deterministic linking methods give equal weight to different types of information a record may contain. For example, a deterministic approach might place equal reliance on a match between the names on two records or a match between two birth dates.

Probabilistic approaches allow the researcher to exploit the probability that a match on particular items is more or less likely to indicate that the individuals named on two records are, in fact, the same individual. For example, birth date information is subject to errors made by a mistake on a single digit, and the number of possible birth dates is relatively small. Names, in contrast, are more likely to be recognizable even if a single error is made. Probabilistic linking allows the researcher to create an approach that weights the value of matches on such information appropriately.

Topic 2: Data Mining

Robert Goerge of Chapin Hall led the session. Goerge asked the audience about their areas of interest in order to tailor his examples. Four of the audience members had health backgrounds. The others represented education, child welfare, and human services in general. A data warehouse was defined as a repository for storing integrated information for efficient querying and analysis. A data mine was defined as a more specific data repository designed for the analysis of particular questions. Bob indicated that the many states were making efforts to put data in one place through a data warehouse and indicated that the focus of the session would be on warehousing individual-level data.

A member of the audience asked for clarification on the definition of a data mine. She thought that a data mine was more targeted than a data warehouse. Bob indicated that it was not only the specificity of the subject of the data mine that distinguished it from a data warehouse, but also the way in which the data are connected. Goerge diagramed a data warehouse. In the example, he depicted a warehouse that contained a master index of all individuals and tables on Medicaid recipients, TANF recipients, foster children, children in special education, individuals receiving childcare subsidies, and immunization data. Goerge pointed out that a common identifier linked each table. He suggested that each table or each topic area could actually be thought of as an individual data warehouse in addition to the entire set functioning as a data warehouse and, by way of example, showed a diagram of a database that Chapin Hall created for the Illinois Department of Children and Family Services (DCFS).

The DCFS Database combines two areas in the child welfare system: abuse and neglect reporting and child welfare services. Goerge indicated that this data warehouse links the two areas by identifying common individuals using probabilistic record linkage. He emphasized that only when one is certain that individual data are linked correctly could one analyze the data. He added that conducting analysis from a full data warehouse is cumbersome and that what is done is to create smaller datasets that contain summary information. These smaller datasets are data mines.

One of the audience members raised the question of confidentiality concerns across agencies. They were also interested in whether or not confidentiality was a concern within agencies. Goerge pointed out that one of the best ways to relieve confidentiality concerns was to emphasize the fact that the data will only be presented in an aggregate format, say in groups of 5 or more individuals, and will not be used for case management. He said that he was not aware of any confidentiality breeches and that because of concerns for confidentiality, some information, such as birthdates, might be excluded from datasets. He also described security procedures to restrict access to data.

Goerge offered examples of agencies that have little trouble acquiring outside data. Child support was the best example of a agency that has been granted access to almost all data by the legislature. Children in foster care are Medicaid eligible so child welfare agencies are granted access to their Medicaid claims data. Child welfare and TANF agencies regularly exchange data in order to determine IV-A funding eligibility.

Updating. To be useful, warehoused data must be unduplicated and up to date. Illinois updates its data warehouse monthly. The decision of how often to update is critical to the effort it takes to maintain the data warehouse.

Geographic identifiers. An audience member asked if the address is used to link records. Goerge said not usually used unless one knows that the address is from the current time period. He added, however, that having some piece of geographic data--such as a county code--improves the quality of the match. To do more small area data analysis, data are geocoded.

Standardization. Standardization is required across databases and how to standardize service events is a major issue. The standardization process involves reconciling the master index and the events across the data warehouse and resolving all inconsistencies before analysis.

Efficiency. Efficient information storage requires information to be stored once in the data warehouse. A second key to efficiency is linking events. For example, within a master index, we want to know then relationship between individuals. A link table that presents the relationship between pairs of individuals would significantly boost efficiency. Linking individuals or events and storing that data in link files is a good way to make queries more efficient.

Summary

A data warehouse is a good way to deliver information to the users and to store information safely. It provides end users with a single data model and a query language. Goerge said that Florida and Ohio had two of the largest data warehousing efforts that he knew of. He pointed out that, in response to welfare reform, many states were interested in linking their TANF and child welfare system data in order to measure the effect of welfare reform on the child welfare system. Many data warehouses are designed specifically for this research. Regarding standardization of database formats, the audience and Goerge pointed out that there is no national center for social program statistics unlike the areas of health and education. It was thought that a national center would benefit the standardization of data definitions and the development of data warehouses.

Topic 3: Small Area Analysis

Fred Wulczyn and John Dilts of Chapin Hall led the Small Area Analysis session.

Geographic Information Systems and Small Area Analysis

Introduction

In his introduction, Wulczyn announced that Asher Ben-Arieh would lead a discussion around issues he raised in his talk the day before. Wulczyn then provided a cursory history of small area analysis, tracing its origin in Chicago to the early days of the Cook County juvenile court a century ago, when the court began to map the addresses of the children coming before the juvenile court. The recognition that many came from the same neighborhoods helped foster an investigation of the ways in which neighborhoods were implicated in child development and that neighborhood flaws, not the character flaws of children and families, might result in children coming to the court's attention. Early geographic mapping was done by hand. A purpose of this session is to cover the process of conducting small area analysis and to cover some of the terminology associated with it. It is also meant to show how maps can be both useful tools and can create pitfalls. Wulczyn said of maps, "As is so often the case, pictures do say 1000 words, but there are at least 1000 words that are unsaid by those pictures and we want to show you both sides of that scenario in our presentation today."

Geographic Information Systems

Dilts set out to discuss geocoding and mapping. He defined geocoding and sketched his purposes, saying:

Geocoding is the process of taking data of any sort and making it analyzable in terms of its spatial characteristics. That sort of analysis can be a visual analysis, the kind you do when you're drawing maps, but it can also be statistical analysis. There are all kinds of techniques that statisticians have used and geographers have used for a long time to analyze data that has geographic characteristics attached to it.

There are two aspects to geocoding. One is the standardization of address records. The other is the process of matching that standardized data to its geographical reference point.

Standardization is necessary because address records are particularly complex and are typically stored as strings of data. Such strings can be difficult for computers matching to interpret unless they are standardized. Steps in standardizing can include separating the elements of the address into units that that can be recognized for matching, like house numbers, street names, or zip codes; by cleaning out extraneous information; and by always using the same abbreviation to represent the same thing. The approach to standardization depends upon the number and quality of records. A few hundred records might be best standardized by hand. But large record volumes or data of varying quality might be best standardized by computer. Chapin Hall uses a standardization program called AutoStand.

Dilts discussed how addresses, once standardized, could be matched with a particular geographic location by the use of geographic reference files, such as the Census Bureau's TIGER ⁽¹⁾ files or other files. TIGER files contain geographic coordinates and Census information for every address within a given area. There are two methods of matching the address to a location--deterministic matching and probabilistic matching. Such desktop packages as Map Info, Arc info, or Map Point perform deterministic matching in linking addresses and geographic records. Data problems can interfere with the reliability of deterministic matching, making probabilistic matching a better choice. Probabilistic matching is done by a specialized software. Chapin Hall uses Auto Match. Auto Match links records on the basis of information that is similar, within certain parameters that are set by the user. It's called probabilistic because the matching is based on an underlying statistical probability that two records are in fact the same record. Once you've done this you can move on to the fun step of trying to analyze your data either visually or statistically.

Questions

Wulczyn asked for questions.

An audience member asked if Auto Match users can set their own levels of probability? Dilts replied:

Yes, absolutely. The software allows total control of the match in terms of the amount of importance that you want to attach to various elements. It might be very important that the house number match exactly, but not so important that street name match exactly. You may allow a little bit of difference in the spelling of the street names that sort of thing. It allows you to determine those levels. It also allows you to determine the overall levels of certainty that you want that the two match.

A number of state representatives described the mapping software in place in their areas and on the difficulties of obtaining the necessary specificity. Wulczyn commented:

On the geocoding side, the lowest level of granularity is the address. Literally, the TIGER file is latitude and longitude coordinate for every address that the census has. And from that point you can wrap it in any kind of political or social subdivision, zooming in and out depending upon the situation. The issue, if you only have zip codes you don't necessarily have to go through a geocoding process, you already have a piece of geographic information. The question is how well does that overlap with political, social, or other types of subdivisions.

Mapping

Like statistical analysis, mapping helps tease out the patterns in the data and make them apparent. With a map, information can by layered by overlaying successive slides containing different types of related information. Wulczyn and Dilts showed a dot-density map of the homes of substantiated abuse and neglect cases. That information was layered on a map of Chicago that contained the boundaries of the city's 77 community areas. Of that map, Dilts said:

You can begin to impose a little order on the data and allow comparisons to be made. But dot-density maps are themselves pretty limited because they don't allow you to . . . quantify comparisons between the areas. You can eyeball this and draw the conclusion that some areas have much less than others, because there's a density clustering around these dots, but that's about all you can say. Given that, the thing that one has to do next is to begin aggregating the data within these areas and to create what's called a thematic map.

Wulczyn added, "The question here is . . . how do you use the maps to tell an effective story? That's essentially one of the things that we're trying to do within the social indicators context is to be able to describe what's happening." Wulczyn then showed how examining smaller geographic areas and by studying the density of incidence of abuse and neglect rates, the user can adjust the way he or she constructs a mental model "about the underlying social dynamics in these areas." He further developed this illustration by showing maps of small geographic areas that included both abuse and neglect rates and rates of AFDC (Aid to Families with Dependent Children) receipt for small areas. He showed how the seeming relationship between high rates of abuse and neglect among families receiving AFDC indicated by the map was an oversimplification.

The presenters continued adding information to the map, explaining how far mapping would take them and noting the point when they reached the "limits to how much insight these visually appealing maps actually impart." By combining the maps with line charts, the presenters matched rates of child abuse and neglect cases with rates of AFDC participation to illustrate the relative levels of incidence, a comparison that helped reveal a weak relationship between high rates of AFDC participation and high rates of child abuse and neglect reporting. Wulczyn said

So what was visually appealing upon closer examination, again the story shifts a bit. It suggests there is something more subtle going on. We have spikes here. The point is that in adjacent areas, adjacent not in a geographic sense, not neighboring Census tracts in a geographical sense, but neighboring areas with respect to how many AFDC participants they have or the rate per thousand, in that sense of a neighbor, they could have very different child abuse and neglect reporting rates. And maps are as likely to obscure that point as they are to reveal the point. So (we must) go back to a more standard X/Y plot... Remember, we started out this series of slides with the traditional X/Y plot . . . . we're not abandoning the need to look at things in a more traditional way because it reveals that there may be problems with imparting too much causality to the relationship between AFDC and child abuse and neglect reporting rates.

A member of the audience said,

If you were looking at Child Protective Services (CPS) and wanted to do an intervention in that area, what this is saying to me is that maybe the common thinking about the relationship between CPS and income isn't holding true. And then you would want to map some other things like foster care, family size, ethnicity, and that would start to tell you more and you would start to change your variables.

Wulczyn replied:

That's right. And that's the advantage, you're essentially creating multiple links to a variety of standardized data sources. And so the Census data will give you some things about household structures, so on and so forth. If you have other sources of administrative data or survey data that you can integrate with this sort of larger schemata of information, it maintains the organization of information so that the process of going through and generating questions isn't delayed because you have to get the information together. It allows you to do things in a very interactive way and change your perspective in a real-time sort of way whereas in the old days it would take a long time to get the information.

During the remainder of the session, there was some discussion of mapping software, including costs and limitations. Ann Segal said that Jake Jacobson of Charlotte, North Carolina was using a variety of mapped data, including foster children's residence, Medicare receipt, bus routes, utility shut-offs, and other information in order to help planning for multi-generational daycare centers that would serve as senior centers and child care centers. In addition to mapping, he evaluated children and used other tools. Segal noted that Jacobson's maps are not public documents, but are used for internal planning purposes, a way of protecting confidentiality.

Topic 4: Revisiting School Readiness and Promotional Indicators

Speaking at this session were

• Mairéad Reidy, Senior Research Associate at Chapin Hall
• Elizabeth Burke Bryant, Executive Director of Rhode Island KIDS COUNT
• Reeva Sullivan Murphy, Childcare Administrator in the Rhode Island Department of Human Services
• David Murphey, Senior Policy Analyst in the Vermont Agency of Human Services
• Beatrice Colón of the Illinois State Board of Education
• David Ayer, Director of Research, Evaluation, and MIS in the Maryland Governor's Office for Children, Youth, and Families
• Monica Herk, Project Coordinator of the Georgia Child Indicators Project
• Janel Harris, Research Scientist in Minnesota's Department of Health, Division of Family Health/MCH
• Steve Heasley, Consultant to the West Virginia Governor's Cabinet on Children and Families

Mairéad Reidy

Reidy began the session by recapitulating some of the previous discussions on school readiness indicators, including some themes of the December 1999 meetings on school readiness measures held in Providence. These included the ambiguity of the term "school readiness" and the need for consensus definitions of what children should be ready for and what schools need to do to be ready for children. She showed a series of slides:

New England Meeting of the Child Indicators Projects: Forum on School Readiness and Childcare Indicators

Hosted by Rhode Island KIDS COUNT

Providence, December 2 & 3, 1999

This meeting was sponsored by the Advancing States' Child Indicators Initiatives project of the Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services, in partnership with the Carnegie Corporation of New York Starting Points Initiative.

Meeting Objective

To assist states and communities participating in the ASPE Child Indicators Project and the Carnegie Starting Points Initiative to develop practical indicators that can be used:

• To track children's readiness for school
• To monitor the childcare infrastructure in terms of quality, affordability, and accessibility

School Readiness

• What children know and can do
• Conditions that surround children
• Availability, quality, and accessibility of services
• Whether schools are ready for children

What Children Know and Can Do

• Physical well-being and motor development
• Social and emotional development
• Approaches toward learning
• Communication skills, language development and cognition
• General knowledge

Physical Well-Being and Motor Development

• Nutrition
• Vision
• Hearing
• Asthma
• Allergies
• Asthma hospitalization rates

Social and Emotional Development

• Can meet and play with different children appropriately for his/her age
• Adapts to changes in routines at school
• Expresses basic emotions appropriately for age
• Develops and maintains friendships to an age-appropriate level
• Separates easily from caregiver
• Uses problem-solving skills to address social dilemmas with peers

Child and Family Connections

• Can meet and play with different children appropriately for his/her age
• Adapts to changes in routines at school
• Expresses basic emotions appropriately for age
• Develops and maintains friendships to an age-appropriate level
• Separates easily from caregiver
• Uses problem-solving skills to address social dilemmas with peers
• Percent of out of home placement
• Percent of children experiencing domestic violence
• Continuity of care givers
• Number of residential moves
• Parents at risk (e.g., alcohol, substance abuse, depression)
• Percent of children with elevated lead levels
• Access to multiple adults

Services Provided

• Access to primary care physician
• Access to breakfast program
• Access to school lunch program
• Access to early childhood programs
• Access to libraries
• Access to preschool programs
• Access to family services
• Access to community resources

Ready Schools

• Offer school-based health clinics
• Offer breakfast programs
• Offer lunch programs
• Offer professional training to teachers
• Offer health screening
• Offer counseling
• Monitor health insurance
• Monitor immunization
• Parent involvement in schools
• School facility is used for non-school activities (e.g., before- and after-school programs; family resource center)
• Teachers have access to school services

Data Collection

Speaker John Love said that:

• Individual-level data is expensive to gather.
• Survey sampling can allow a relatively small number of cases to reflect the diversity in the community.
• It is important to identify priority groups that can be asked "cross-service"/"cross-discipline" questions about children.

Communications Strategies

Love recommended that we:

• Use a marketing approach to present data. Couple data with human presentation (e.g., if reporting results of survey of kindergarten teachers, have a panel of teachers share their stories at the same time). Explain data in terms that resonate with the audience (e.g., link turnover rates in childcare to impact that turnover has on the business community).
• Recognize that change makers may need 100 charts; have different datasets that could be used to address your changing needs and theirs.
• Sustain a core of indicators to be tracked consistently over time.
• Apply the findings to continuous community improvement efforts.

Data Interpretation

At the New England meetings, Love suggested states:

• Focus on the collective or aggregate status of entering kindergartners.
• Use community-wide estimates of school readiness.
• Develop estimates of readiness for all important subgroups within the community

Childcare

• Quality
• Affordability
• Accessibility

Childcare Measures that Could be Examined Using Existing Data

Quality

• Percentage of programs accredited
• Teacher turnover
• Education level of teachers

Affordability

• Percentage of families spending an identified portion of their incomes on childcare
• Value of subsidies in comparison with market rate

Accessibility

• Number of subsidized slots
• Number of licensed slots

Elizabeth Burke Bryant

Bryant said that the state of Rhode Island is very serious about developing school readiness indicators because of the major investments the state has made in childcare during the past two years. Legislative leaders will continue to support the expansion and improvement of Rhode Island's early care and education system if they have evidence that these investments are having a positive impact in terms of children arriving at school ready to learn. Because time was of the essence, the Rhode Island Child Indicators team decided that there is no perfect set of measures, and we moved ahead in selecting a logical set of indicators that are a good reflection of a young child's health, cognitive ability, and social competency. Some of these indicators were already being tracked by Rhode Island Kids Count; others had to be developed and put in the field. Rhode Island added childcare quality questions to our market rate survey of childcare providers (now in the field), and we are adding school-readiness questions to our survey of kindergarten teachers. A complete listing of Rhode Island's school readiness indicators was distributed as a handout for the workshop participants.

Reeva Sullivan Murphy

Murphy said that her background as an early childhood educator and kindergarten teacher served her well in her new position as state Childcare Administrator. As part of her work with Rhode Island's Child Indicators Team, she has been developing indicators that, taken together, paint the full picture of a child's readiness for school, and the school's readiness for each child. It is the combination of good health, and a comfort level in interacting with other children that make some children more ready than others in a kindergarten teacher's view. Murphy said that using the market rate survey every two years will be a great way to obtain data on school readiness, and that the willingness of the people who administer the comprehensive School Accountability for Learning and Teaching (SALT) ⁽²⁾ survey to every public school student will provide the kindergarten data that we need. She encouraged other states to select a list of indicators that cover the key areas and that will make sense to policy makers when it comes time to report on whether the state's early childhood investments are making a difference.

David Murphey

David Murphey of Vermont picked up on the idea of consensus in the area of school readiness. He said that over the past year, a degree of consensus has emerged among those working on school readiness indicators and discussed some of the different approaches he sees states taking in understanding school readiness. Murphey saw consensus around these ideas:

• Multiple perspectives should be acknowledged and multiple measures are appropriate
• Even when focusing on the child, school readiness is a multidimensional construct
• School readiness is not a test for children or teachers
• Readiness is an outcome and is conditioned by prior experiences and serves as a predictor of how well a child is going to do
• There is value in both reporting on readiness in the aggregate and using individual-level data for some purposes, as long as the appropriate protections are in place

Murphey identified these as different paths states are taking:

A

• A checkpoint at a key developmental transition and thus an important point for analysis
• A profile of community performance
• A tool for community systems improvements

B

• A continuous assessment process
• A profile of student performance
• A tool for improvement of instruction

Murphey discussed Vermont's approach to assessing school readiness.

Overhead ⁽³⁾  A Chronology of Vermont's Strategy: A 3-Part Assessment of School Readiness

• Brief, child-focused questionnaire completed by the kindergarten teacher (5 minutes).
• A school-focused principal questionnaire on ready schools covering such issues as parent involvement, the quality of the relationship between the school and the early childhood provider, what supports and services are available to kindergarten teachers. It is six pages in length and is expected to take about ten minutes.
• A very brief child-focused health checklist. This instrument has eight questions on such topics as hearing, vision, immunization, dental health care, developmental problems, and health insurance.

Vermont has used focus groups of interested parties to receive input on this plan and to obtain information on ways to revise their instruments. This fall, Vermont plans to use this assessment statewide. This survey will replace an earlier survey that the state had found less useful than it hopes the new assessment will be. Murphey made draft instruments available to the audience. In response to questions from the audience, Murphey said that they plan on asking teachers questions on their own experiences and history. He also described the anticipated ways in which the Vermont data collection would be conducted.

Before Beatrice Colón began, Martha Moorehouse said that Rhode Island's initiatives in the area of school readiness are motivated in part because they are making big investments in this area. She asked if the work in Illinois and Vermont was taking place in the context of changing investments and the desire to track those investments. Murphey said that there is strong support in Vermont for standards-based assessment but that it remains to be seen if there will be a funding investment.

Beatrice Colón

Beatrice Colón said that about six years ago, the State Board of Education decided that teachers needed a systematic way to collect and assess data on what children should know and what they do indeed know. A small group of teachers and administrators developed a system to use work sampling as an alternative to testing. The system has three components: developmental checklists, portfolios, and summary reports done three times a year. These reports provide a chance for the teacher to reflect on what the child has gained. They can be shared with the parents, too. Colón pointed out that Illinois has 450 prekindergarten programs. Not every one of those programs is using work sampling. Each district can choose what kind of assessment they will use. At most levels, the work sampling system is being implemented using only one to two domains as a start. A few systems are using all of the domains.

The developmental checklist follows the domains. Those domains are:

• Personal and social development
• Language and literacy
• Mathematical thinking
• Scientific thinking
• Social and cultural awareness
• Art and music
• Physical development

Some of the systems that tried to implement work sampling using all domains had a difficult time doing so.

Colón said that an integral part of the system is continuous training and staff development. It is her view that continuous staff development must be built in to the process and school administration must support it. Without those steps, the system will fail. Because there are 450 prekindergarten programs (excluding Chicago) and more than 400 classroom programs in Chicago alone, Illinois State Board of Education (ISBE) contracts out some of this staff development work. A challenge to work sampling is the variation in the ways in which principals implement it and support, or fail to support, staff development.

As a result of the work sampling, and of questions and suggestions from the school district, ISBE has developed and is further developing indicators, outcomes, and standards for early childhood programs. Currently, Colón has a draft document that is being reviewed. They are also developing program standards for birth-to-three programs.

Colón said that Illinois does not collect data, per se, on work sampling. But ISBE does collect data on children in the prekindergarten program. There is a record for each child and family that contains health, safety, education, and other data. There is also a longitudinal study that follows a sample of children through 12^th grade, but that the population of interest is very transient.

David Ayer

David Ayer said that, in the mid-1990s, Maryland developed its Maryland Model for School Readiness. They wanted a way of assessing children by the time they reach the end of their kindergarten year. The work sampling system, which covers a great deal of ground, was chosen to do that. Since 1997, there has been a pilot implementation underway. All of the twenty-four school systems in Maryland will use this in kindergarten in the coming school year or the one following. They will look at all of the domain areas available in the work sampling system. Maryland will examine the aggregate percentages of children at the end of kindergarten reaching a proficient level. This information will be incorporated into Maryland's indicators.

Monica Herk

Monica Herk identified four Georgia benchmarks. These are:

• Percentage of low-income students in Head Start or prekindergarten programs.
• Percentage of kindergarten students who attend preschool or childcare programs. (Georgia has universal, state-funded prekindergarten program. This eases data collection because one of the mandates accompanying funding is that the preschool programs, even if they are in private settings, help children make connections with public schools.)
• Percentage of kindergarten students passing the Georgia Kindergarten Assessment Program (GKAP).
• Percentage of students who are two or more years overage in third grade.

Herk focused mostly on the GKAP, created by legislative mandate to assess the readiness of kindergarten students to enter first grade. Herk provided eight pages of background information on the Georgia Benchmarks and on GKAP. Summarizing the GKAP findings over the past four years, she said that the numbers have been relatively unchanged. (A handout put this number between 87.3 and 88 percent over the period.) She said that the Georgia example shows that, although the state has made a huge investment in prekindergarten education, the measure doesn't show a change. However, evaluation studies that have tracked children are showing that children who go through the state-funded prekindergarten program seem to score better on the kindergarten assessment.

With the most recent revision of the GKAP (GKAP-R) some of the earlier data will not be comparable to later data. One of the changes made in this revision aligned the GKAP-R more closely with the Georgia Quality Core Curriculum ⁽⁴⁾ content standards. This alignment will also mean that Georgia's scores will not be comparable to data from other states. There have also been changes in methods of administration in an effort to reach a form of administration best suited to children of this age.

The GKAP-R has three windows of assessment, one at the beginning, one at the middle, and one at the end of the school year. This supports the use of the GKAP-R as a diagnostic tool for the teacher.

Janel Harris

Janel Harris said that the February 2000 meeting on promotional indicators held in St. Paul had tried to address three goals:

• To explore the conceptual framework for promotional indicators;
• To obtain consensus among the states, especially the four convening states--New York, Georgia, West Virginia, and Minnesota--on particular promotional indicators that they could agree upon as being important and also possibly useful to collect data on; and,
• To get information on the federal perspective on opportunities to incorporate promotional indicators into the federally mandated indicators.

Harris felt that they made progress on all of these goals, but that the promotional indicators effort remains very much a work in progress. One conclusion they drew was that they needed to further examine the research that underlies some of the areas in which indicators are needed so that they could see what research relates to those things that seem intuitively to have value as promotional indicators. Harris also commented on two talks on indicators by federal agency staff--Martha Moorehouse of ASPE and Casey Hannon of the CDC. Harris felt that most of this federal work was deficit-oriented and she found very little in the way of positive or promotional indicators. She said that, none the less, she felt it was important to push forward the promotional indicators agenda, in part because people are "really tired of being viewed as a collection of problems." Promotional indicators can provide a more balanced picture of the situation being examined.

Steve Heasley

Steve Heasley went over what has happened with promotional indicators since the Minnesota meetings. He said that it is important to point out that a lot of the interest in promotional indicators stems from frustration with traditional indicators that are deficit oriented. He said that it is interesting that frameworks are developed "to measure, 'child well-being,' but, because of the limitations of the data, what we are measuring is the absence of well-being." The philosophy before promotional indicators is to provide measures that do capture well-being--those that are related to strengths and assets.

Heasley said that he was struck by Ben-Arieh's presentation and the work on the indicators worldwide.

Some of the areas in which promotional indicators could be developed include:

• Are children being read to?
• Do children have caring adults in their lives?
• Measures of parenting skills
• Volunteering in the community
• Supporting family life

Heasley said that the cross-state email group is open to anyone who wants to participate, not just to the four states that convened the February meeting.

Part 2

Topic 1: Opportunities and Difficulties in Large Urban Survey Research

Speaking at this session was Kristin Shook of the Northwestern University Institute for Policy Research. Shook is the manager of the Illinois Family Study. Shook began by introducing herself and naming some of her collaborators on the Illinois Family Study, including Principal Investigator Dan Lewis of Northwestern, Stephanie Rieger of the University of Illinois at Chicago, Paul Klepner of Northern Illinois University, and James Lewis of Roosevelt University.

The Illinois Family Study (IFS) is a six-year longitudinal study of approximately 1,500 families that were receiving Temporary Assistance to Needy Families (TANF) in late 1998. Most had at least one child younger than 18 years. The IFS looks at a variety of different outcomes, primarily related to the grantees. In most cases, the grantee is the mother of the family. The issues and the purposes of the IFS are to explore employment trajectories of women receiving welfare as they transition off of welfare. The IFS, in part because of Shook's interests, also includes a number of questions that focus on children. Shook noted that the legislation that mandates the ISF says that it is not to be funded by the Illinois Department of Human Services. It does receive some funding from other state agencies and from local and national foundations. The possibility of federal funding exists.

Shook said that "researchers and state policymakers often come at the questions from a slightly different angle" but that the IFS was a unique opportunity because the state Department of Human Services is very interested in understanding what about welfare reform is working and what is not. IFS has an advisory panel on this study of key state legislators that gives direction about the kind of information that would be very useful for them in future years and the project has a good working relationship with DHS, allowing them access to the kind of data systems they need. Other agencies and advocates are also very involved.

Earlier Surveys

Shook noted that the IFS team has also relied on existing surveys related to welfare research. In particular, she mentioned:

• The Three Cities study--a panel study of welfare recipients--done by P. Lindsey Chase-Lansdale of Northwestern
• The Women's Employment study at University of Michigan by Shedon Danziger and a large group there
• The New Hope Project also conducted at Northwestern by Greg Duncan

The measures and survey tools used by these studies helped the IFS team develop their measures, allowing for comparisons and for some certainty that the materials and measures had been tested.

Administrative Data

Shook also noted the that they were able to link their survey data to existing administrative data from Illinois human service agencies and that they had worked with Chapin Hall to develop their ability to use and understand the administrative data.

Child Well-Being Indicators

Shook described how she, working with Northwestern's Duncan, had included indicators of child well-being into the survey. She and Northwestern pediatrician Dana Hall found support from the National Institutes of Health (NIH) for a supplemental study on child neglect. This supplemental study will involve 500 families from the original sample who had children younger than three years at the time of sampling.

Shook was asked, "Is your expectation that there will be a high level of various kinds of neglect?"

She replied that work already underway in Illinois using administrative data had examined the links between child welfare and cash assistance systems prior to TANF. Some of that earlier work had shown "that about 5 percent of a cohort of AFDC recipients prior to TANF became involved in the child welfare system over the course of the year. And that was any form of involvement, an indicated report, a case opening with child protective services." Based on these and other estimates, she expects:

. . . approximately 75 families a year that had some involvement with the system. And over time the cumulative involvement gets as high as 25 percent over a five-year period. So with the 1500 families we did some power analyses to make sure to get the results we are looking for, and also with the 500 families as well.

They are also interested in changes in Medicaid coverage--even temporary losses of coverage--and changes in health care and medical neglect rates. She continued:

The advantage of looking at these links within the context of welfare reform is that there is variation produced in each of these independent variables I described that is automatically being produced by welfare reform policy. So we are taking advantage of that variation to explore these links to different forms of child maltreatment. And given that in Illinois about almost two thirds of all the kids who enter foster care in a given month come from a family that recently received welfare, exploring this within group variation was also important to us. So often you see poverty or welfare used as kind of a predictor for child protection intervention, but nobody really understands what that means.

The research will also explore what subgroups of the population are having difficult times.

Shook then moved to a secondary goal of the supplemental child well-being study, to carefully define what they mean by child neglect and different forms and child maltreatment. She said that there are three sets of outcomes that the researchers are interested in:

The first kind of precursor set of outcomes has to do with severe risk. Situations we can identify that are likely to present a potential harm to children. So this is beyond being a single-parent family or something like that. It's the presence of multiple hazards in the home, being without health-care coverage for a significant period of time.

There is a second stage that looks at harms to kids--accidents, injuries, poisonings. The things we link back to medical records, information from children's pediatricians, and health-care practitioners that they see to try to get a better understanding of whether this is a pattern or is this something that just happened once. We want to make sure that we're kind of separating out things that happened normally and something that is likely to be a sign of more problematic family functioning.

And a third level is looking at how these different sets of outcomes relate to child protection intervention. So often in child welfare research the outcome used as an indicator of child abuse or neglect is involvement with the child protection system.

When asked how long the supplemental study will take place, Shook replied that the NIH funding is for five years, but that they plan to:

make sure we get enough baseline information that it will be convincing to funders later on. We plan that to go all out during the first year at least to make sure we really get a good set of baseline indicators. Given that we have all this access to these other systems, in recognizing that this is a really unique opportunity, I have to be strategic about convincing people about this. It's interesting, in groups like this people understand that. But a lot of times people say, for example when we go to foundations, "Oh, this is a study and deals with welfare, we've funded enough of these welfare studies at this point." So it's really been more of a struggle than I thought it was going to be. I thought it was quite clear about how unique this opportunity was, but apparently not to everyone. But we're getting there, it's just going to take a little more time before it is not our main focus all the time.

Earlier in her talk, Shook had said that one of the project's goals was obtaining a careful definition of neglect. At this point, she elaborated on that, saying:

. . . there are some real issues in identifying neglect. When we say we are identifying neglect, we are asking questions in a retrospective way. We are relying on combinations of indicators so that in the context of the interview you're not identifying neglect based on the answers to one of the questions or an obvious combination of the questions. We are not just relying on combinations of indicators for that purpose, we set this up theoretically in terms of the way you can go about identifying treatment and ways you can't. But you need to use combinations of questions not just one question to really see what's going on. And we're also relying on the medical records very heavily, which are based on past interactions with health-care professionals. And there are lots of conflicting feelings about whether we need to err on the side of figuring out that something is going wrong in the family and then reporting. Or realizing that so little is understood about the etiology of neglect we have to keep that in mind as we're tracking the families over time, and not set this study it up in a way that kind of subverts its own purpose right from the get go. I don't know if this is really the right time to get into a discussion of ethical issues. I'd be glad to. I haven't quite decided where I fall on the issue. So the physical abuse, a lot of the questions we're asking would be about punishment strategies. Things based on the common tactics scale. But we're not asking a lot of the more severe items because we learned in pilot testing that these were very insulting to a lot of families. I myself did not feel comfortable asking a lot of these really severe questions about punishment techniques. So we probably have enough in there to begin to identify some kind of a threshold but not get at the extreme end of the continuum.

An audience member asked: If you have that administrative data you would be able to track allegations or people who are at least classified by the system. Shook suggested that:

Some of this might come up in medical chart review too. We might detect some suspicions that physicians have about abuse. But the neglect indicators are less problematic because, for instance, the environmental neglect indicators are about food insufficiency using the U.S. Department of Agriculture's Food Insufficiency Scales. We gear them really towards children's experiences, more so than adults' experiences, of food insufficiency. And we have different levels of risk just like the U.S. Department of Agriculture does. We have mild, moderate and severe food insufficiency. So we try to kind of set up our indicators in a way that there's more of a continuum and we are not just kind of looking for some threshold yes/no but exploring this in a little more of a complex way.

Respondent Permission to Access Medical Records and Other Administrative Records

An audience member asked how the IFS team got permission to use medical records. Shook explained that respondents signed two consent forms, one giving permission to do the study and the second granting permission to access information from "a long list of administrative data systems, school records, medical charts," and other datasets. This avoided going back to families later on. Shook reported that about 85 percent of respondents gave their consent to access the data during the first wave and they're hoping to boost that during subsequent waves as trust grows. (Later, Shook said that she thought state entities did not need permission to examine administrative data in this way, but as a private consortium of researchers, the IFS team felt like they did need respondent permission.)

An audience member suggested that families connected with protective services might be reluctant to grant permission to access records, creating bias. Shook explained that they could assess such a bias because they drew their sample from administrative data and can draw a similar sample for comparison.

An audience member asked if the study collects data on household composition in order to examine the relationship it might have with good or bad outcomes for children. Shook said that they were getting at this by using household rosters and asking about the relationships among everyone living in the household.

In response to a question, Shook said respondents are paid $30 for each annual interview.

Data Collection

Locating

The IFS sampling frame included addresses, telephone numbers, and information from DHS records back five or six years. All this information, plus the increasing use of the Link card (an electronic debit card for benefit transfer), eased locating tasks. Also, addresses for families receiving childcare subsidies tended to be more accurate than were some other records. Despite this, locating difficulties have slowed the first wave of the study. In response to a question, Shook said that those respondents who are found to have moved out of state will be interviewed by telephone using a shortened instrument.

Scheduling & Refusals

Shook described the challenge of scheduling interviews and the care the staff take to treat respondents nicely. Regarding hard versus soft refusals, she said:

We find that probably seven times out of ten people will say no on the initial phone call and when you callback even just the next day, "Oh, yeah, come on out." So you have to be good at learning the distinctions.

In-Person Versus Telephone Survey

Shook described two limitations of collecting data over the phone. First was the need to use a shorter instrument. Second is that respondents often lack consistent telephone service.

Interviewer Safety

A participant asked about safety for the interviewers. Shook said that she and her team had relatively little difficulty and she discussed some of her experiences collecting data in high-crime areas. In response to a related comment about the difficulties one survey had using school teachers as interviewers, Shook recommended using social work students.

Nonresponse

For the IFS, locating poses a greater challenge than refusals. They have a high response rate among people they actually reach. Among the strategies they have adopted for finding people are canvassing neighborhoods on foot. They also looked at additional databases. In response to a question, she did say it was her sense that those who were really difficult to find were more likely to be not receiving assistance. There was discussion then about who might be missed both in this study, and in other studies of human service recipients in times of transition.

Qualitative Interviews

Shook closed by noting that the IFS plans to conduct qualitative interviews with a 10 percent subsample of families who have multiple involvements with the human services system to see how they experience those involvements.

Topic 2: Data Standardization

The Data Standardization session was led by Allen Harden of Chapin Hall. Harden said that the session might be better named data "decomposition and standardization," because you can't standardize data without thinking about how to break it apart, and noted that the aim of the session is to show how to standardize data for purposes of comparison.

Introduction

Measurement. Using a variety of illustrations, Harden addressed the need to devise measures suited to the topic of interest. When thinking about constructing measures, it is important to recognize early on what you are measuring, that is, what is the unit of analysis. In this work, it is often a child, but might be other things, such as attributes of families or places. Sometimes a statistic, an absolute magnitude, is all a manager cares about. But, this is rare. It is more common for the policy community to need more information, such as information for comparison. Comparisons can require rates, means, unduplication, and other adjustments of data.

When making comparisons among data, it is important to understand the definition of the terms. Harden used kinship care as an example, noting that different state definitions of kinship care inhibited comparisons of foster care population statistics. He pointed out the importance of interviewing the sources of the data to ensure that you know what generated the measure, how it is defined, and who is included in the counted group.

Population of reference. Along with the measure, it is necessary to define the population of reference, the denominator of a proportional measure. A simple percentage is the share of children with particular attributes. That prevalence statistic, may, by itself, be sufficient.

But trying to get information on what produced the statistic in the first place requires a more complicated approach. This is especially true if the numerator is an event or something else that is countable, then it can be important to define the denominator as the population that can be appropriately be thought to be at risk of experiencing this event. When we can define an event and the likelihood of that event occurring to a population of interest, this is an incidence statistic. (Harden noted that because the language used to define these measures comes out of public health and epidemiology, it is negative sounding, including phrases like "at risk." He noted that one can be "at risk" of success.)

Using the example of child welfare, Harden said that the initial prevalence statistic can represent two things--rate of entry (or incidence) and also duration in care.

Comparisons. Harden moved through an array of comparisons to which measures might be put. These included over time, by age of population, by region of residence, by gender, and others. He pointed to the work done by demographers on fertility as an example. A crude measure is the birthrates for a particular populations, more informative are age-specific birthrates that get at the timing of births.

An audience member said that the idea of standardization was attractive, but much of the data he encountered was compromised. He asked for guidance on how much compromise was too much. Harden said that these determinations are made based on what the data user ultimately wants to know, when he or she needs to go out and get better information. He also noted the importance of attaching the right explanations to the data so that there is no confusion about what the data can and cannot explain. Harden showed some comparisons of child populations in Chicago.

Discussion

Discussion revolved around appropriate uses of data.

Topic 3: Interpretation of Data

Leading this session were Robert Goerge and Bong Joo Lee of Chapin Hall. It covered:

• Point-in-time versus longitudinal analysis
• Ecological fallacy
• Small cell size
• Adjusting rates with individuals and communities with regression
• Percentages versus rates in populations

Point-in-Time Versus Longitudinal Analyses

One way to look at foster care is through point-in-time analyses. Another way is to look at the history of an entire entry cohort. Each approach gives different information. The latter approach will reveal time spent in care and show the distribution of care over time. The longitudinal approach is useful in examining what effects the length of time spent in care and point-in-time analysis can overrepresent the number of children who spend a long time in care.

Ecological Fallacy

Using aggregated data to make inferences at smaller geographic levels can be complex. Bong uses the example of infant mortality rates across geographic areas. By examining the relationship among key risk factors and infant mortality rates, Bong can account for about 1/3 of the seeming geographic variability in those rates. By considering such factors as the utilization of prenatal care, more of this variability can be explained.

Small Cell Size

The small size of populations in some geographic areas means that random events can produce substantial variation in incidence, service use, or other statistical measures applied to those areas. One way to compensate for small cell sizes is to simply analyze larger populations. Another possibility is to combine cohorts across years to create a multiyear moving average that compensates for the possibility of extreme variation.

Adjusting Rates with Regression

Lee also discussed the use of regression analysis to help untangle the influences on different variables in a community. In the discussion that followed, Lee and the participants covered how regression analyses can be used in planning and explaining what data show to communities.

Special Session

Because of the widespread interest in the international indicators work of Asher Ben-Arieh, a special session was scheduled in which those interested could ask him questions. Ben-Arieh began by noting that the researchers working on this international effort--about 25 in all--are involved in this in addition to their regular work. He also said that the international working group did not pay anyone's travel or lodging to these five-day meetings.

Ben-Arieh said that the northern European nations are progressive on many children's issues and that New Zealand offers very innovative programs and research. He noted, in particular, New Zealand's antipathy toward achievement and, in agreement with a member of the audience, noted that the New Zealanders like to measure systems rather than individuals. Ben-Arieh also singled out the Australians for their innovative research on the cost of children.

In response to a question about why Ben-Arieh preferred the terms "positive" and "negative" indicators to "promotional" indicators, Ben-Arieh said that the idea of "promotional" indicators, like the idea of outcomes, characterized children as part of a process.

Endnotes

1. Produced by the U.S. Bureau of the Census, TIGER (topically integrated geographic encoding and referencing) files are a digital database of geographical features and political boundaries used to create maps. For more information, see the website of the Census Bureau, www.census.gov.

2. On its website, the Rhode Island Department of Elementary and Secondary Education calls its School Accountability for Learning and Teaching (SALT) program "a school-centered cycle of activities to improve school and student performance in the Rhode Island public schools." An annual survey of students, parents, and teachers helps inform the development of school improvement plans. For more information, see http://www.ridoe.net/default.htm.

3. This overhead is reproduced from notes taken by a staff member and may differ from Murphey's original.

4. Georgia's 1986 Quality Basic Education Act mandated the development of a uniform core curriculum called the Quality Core Curriculum. The Quality Core Curriculum must be included in the basic curriculum provided to students in Georgia public school districts. For more information, see the Georgia Department of Education's Georgia Learning Connections homepage, www.glc.k12.ga.us/qstd-int/homepg.htm.

Session 3: From Indicators to Outcomes: Considerations and Strategies for Communities (May 4, 2000)

Speaking at this session were Ada Skyles of Chapin Hall, Arlene Andrews of the Institute for Families and Society of the University of South Carolina, Michael Bennett of the Egan Urban Center of DePaul University, Jennifer Jewiss of the University of Vermont, David Murphey of the Vermont Agency of Human Services, and Mary Nelson of Bethel New Life Inc. Skyles then recapitulated the previous two sessions, listing a number of key points.

• Public engagement is an active process and you never quite arrive, but must continually engage.
• Different strategies are required for different audiences, but even within each audience, there are a variety of actors.
• It is important to be realistic about public engagement. For example, it requires money.
• Think continually about what is useful. Different audiences may provide different opportunities.
• When you engage the public, you are dealing with a power shift.
• It takes time to build up an infrastructure that allows for community input, but such an infrastructure is well worth the investment if done well.
• Among the concerns presented by states during previous meetings included minimizing the potential of data to stigmatize communities by emphasizing positive change and data that highlights strengths.

Arlene Andrews

Arlene Andrews said, noting that it was obvious, that the only reason to collect data is to use it, particularly in the communities, where children and families live and where change takes place. She said that Institute for Families and Society serves as a broker between providers of statistical data and community users of statistical data. Much of the work of the Institute is with community-based organizations (CBOs). Currently they are involved with 130 organizations.

Andrews said that the Institute has a community assessment program that is largely focused on rural areas. She characterized rural poverty in the south as exhibiting racial segregation and the rural community-based organizations as being generally grounded in African-American culture.

Discussion moved to the value of statistical information and gaps in statistical and mathematical literacy and technological expertise among stakeholders. Andrews used a series of overheads to illustrate how statistical data must fit with the needs of the community and address what communities know about themselves in order to be valuable. She also covered the differences in state and local perceptions of the meaning of particular data and the ways in which data can be used to reinforce bias or become a part of larger issues of power and control.

Discussion

Andrews and the participants stressed the importance of statistical literacy among people seeking to use data on communities. It was noted that CBOs often lack the resources to purchase up-to-date computers, making it difficult for them to effectively use statistical information.

Observations

Andrews offered a number of observations on the use of statistical indicators.

Indicators need to be considered in the context of community life and in terms of specific change processes already underway. (Assume that the community is already working on the problem even if they don't describe the problem using statistics.)

Take into account the ongoing change process in a community. Most communities are organized in some way and those community structures are frequently working with other organizations. Statistical information becomes part of this process, along with qualitative data.

Once you have the data--talk about it, analyze it, try not to jump to any conclusions about it. What are the stories behind the data? (Or what is the data behind the story.)

When you go into a community, get a good sense of where they are and also work to understand what an indicator says (and doesn't say or doesn't capture) for a community perspective.

Issues of power are hypersensitive at the community level. It is critical to be prepared to process data in ways that take into account community sensitivity.

There are multiple venues for the use of indicators.

Data can cause bias. One way it does this is because it seems to create priorities that may be at odds with a community's perceptions of its needs.

One of the challenges to community planning is that people don't know how to use statistics.

Handout

Andrew's handout is reproduced below. She said that the set of questions in the handout are all intended to prompt a group reflection and consensus building or, if there is dissention in the group, then get that documented about what these data could be saying.

From Indicators to Outcomes: Considerations and Strategies for Communities

ASSUME: An inclusive community planning group has a comprehensive set of statistical indicators about children and their families. They have qualitative information to enrich the data. They have measures of dispersion as well as central tendencies, information for subpopulations, and time trend analyses. They have a pretty clear picture of how children in their community look.

What is a community to DO with this information?

A critical principle in the use of community needs and resource assessments is that groups should hesitate to jump to solutions. Groups should carefully analyze and interpret the data through extended discussion. The story behind the data becomes the foundation for exploring relevant and effective solutions.

Community groups who review and discuss the data should consider a number of focused questions to guide their deliberations, such as these:

1. What is the purpose of the data review? Why are we doing this? What does the group hope to accomplish by examining the information?

2. Is the group satisfied that the data is accurate, inclusive, reliable, and valid?

3. Item by item, consider these issues:

1. Examine detailed reports regarding how many children are in the categories of response to this item: how severe is this issue?
2. What does the data indicate about children's strengths? Vulnerabilities?
3. What are the standards by which the group assesses the indicator (e.g. comparison to another community or normative group; community opinion; trends over time)? Standards may be relative to an absolute level (e.g. no drug use is acceptable) or relative to other groups (i.e. this seems typical and appropriate for children this age).
4. When current data is compared to past data and to, trend analysis over time, is the indicator suggesting child well-being is getting better or worse? What are the possible reasons for the status and trend of this indicator?
5. If the indicator or set of indicators suggest an asset that needs to be enhanced or a problem that needs to be reduced, how can the issue be clearly defined?

4. What are the linkages among the various indicators? How are they "woven" together?

5. What is missing in the picture portrayed by the data? What can be done, if anything, to clarify the picture?

6. Taken as a group, which indicators suggest the need for community action? Which is highest priority? Why?

7. Based on this data analysis and interpretation, what courses of community action seem to be appropriate? What is the logic behind the alternative courses? What is known about effective practices?

8. Which course of action is most likely to lead to the group's vision of child well-being in the community?

9. What is the consensus about the most appropriate and feasible course of action?

After communities review their child well being indicators, the next step is to develop an action plan to promote community resources so that children's well-being can be maintained or enhanced. The plan leads to implementation, evaluation, and reassessment.

Andrews and the audience discussed how issues in a community defined, whether by the community itself, by the state or other actors, or both, and participants outlined ways in which they helped explain data to communities and make data useful. These included

• Providing technical assistance or money to buy computers
• Being careful to never put out data without also putting out an explanation.

These and other steps were thought to guard against misinterpretation of data. One participant advised convening a focus group before data release in order to increase the possibility that all possible reactions would be anticipated.

Using the World Wide Web

Andrews asked participants how they encouraged community-based organizations to use the World Wide Web. Answers ranged from walking them through the use of the web in tandem with making applications and other useful materials available thorough the web, and providing an 800 number that organizations could call to get information otherwise available on the web.

Logical Leaps

Ways of countering inappropriate leaps of logic by data users were discussed. One recommendation to avoid this was in careful framing of data and careful statement of the definitions behind the data. Andrews and participants volunteered examples of differences between the way data elements might be defined by those who collect them and by communities.

David Murphey

Background

David Murphey said that the Vermont Agency for Human Services has helped to establish 12 regional partnerships in the state. These tend to be collaborative partnerships involving the education, health, and other agencies along with nonprofit organizations. These collaboratives meet on a regular basis. They are responsible for implementing outcomes at the community level. These partnerships are at different stages of development.

Vermont's approach embodies these steps.

• Share data
• After outcomes are agreed upon, put out the data related to those outcomes.
• Communities want to know what works and part of the state's job is to take on the task of showing communities what programs and strategies have evidence of effectiveness. Murphey noted that responses from communities might be that "nothing works" or that "what we've been doing for a long time is working fine," such assertions may lack the confirming data. Part of the state's mission is to help show communities the programs likely to be successful and which ones don't yet have confirming evidence.
• In a process akin to professional development, increasing people's comfort with an outcomes-based approach, including increasing statistical literacy through the use of training sessions. Sessions of that type are described below.

Overhead

Murphey displayed the overhead below and commented on it, calling it a work in progress.

Vermont: An Evolution of State-Regional Partnerships

Trainings to-date sponsored by the Agency of Human Services in statewide meetings:

• "Growing Great Groups:" Dynamics of group-building and sustaining (leadership, decision-making, uniting diverse partners and similar tasks.)
• Families as Partners (family-centered practices, treating families as partners with expertise).
• "From Risk to Resiliency." This is a training piece related to risk and protective factors, including the idea of what counts for individual and community-level well-being, with a focus on the more resilient or more positive definitions.
• "From Promising Approaches to Best Practices." This is an attempt to summarize in a clearly digestible format, without pretending to be an encyclopedia or the last word, what works.
• Using Community Data. This includes how to do a community assessment, how to find off-the-shelf existing data, and the difficulties of generating or collecting new data (particularly by surveys). The training also includes some basic statistical concepts, including the problems of small cell size, the role of chance, and how to use data to set some strategic priorities.
• Strategic Planning, including how to use data as part of a more comprehensive process and how to use them to set priorities.
• State-Local Negotiations. If funds are saved through the use of community programs, some of those funds can be retained by the communities and reinvested in programs the community thinks successful.
• Results-Based Budgeting. This relies on the model developed by Mark Friedman.
• The "Patch" Concept of Service-Delivery (with Gerry Smale). This concept was developed in the U.K. and it is an integrated service-delivery model, based in a community, that relies on paraprofessionals and people who are trained in multiple disciplines so that community members have one-stop shopping when it comes to receiving services. (This tends to work best, in their experience, at the intersection of two service districts in which participants need to choose one or the other.)
• Asset Based Community Development. John McKnight (Northwestern University) presented on his work with a national research and training program focused on neighborhood revitalization, as part of a day-long conference on "Building Community Through Citizen Participation" (Nov. 12, 1998).
• Living Systems. Margaret Wheatley (author of Leadership and the NewScience, and others) presented on her work with organizations aiming to achieve coherence in the midst of chaotic environments, as part of a day-long conference on "Getting the System to Move" (Jan. 25, 1999).
• Local Economic Development. Michael Shuman, consultant, and former co-director of the Institute for Policy Studies, presented on achieving local self-reliance in a global economy, as part of a day long conference on "Partnering for Sustainable Development" (April 12, 1999).

Murphey concluded by saying that his presentation was designed to make clear the range of things that Vermont has shared with community partners. Most recently, with the help of the Casey Foundation, they have provided funds to the partnerships for technical assistance, and provided a menu of training options from which to choose.

Jennifer Jewiss

Jewiss was involved in a qualitative study of two of the community partnerships in Vermont mentioned by David Murphey. The designers of the study chose to examine two partnerships that showed progress on their indicators, without saying necessarily that the partnerships were causing this improvement. The study yielded two findings. One was that data become the most useful when viewed as revealing the situations people experience. The second was that it is important to work with the media to help them understand what the data say about positive change.

The Data & The Stories Behind the Data

The communities involved in the study reported that having the quantitative data on child well-being was very important to them and that the fact of having the data had a powerful impact in discussions facilitated by local leaders.

However, in addition the research team heard that the local communities needed to find other effective ways to relay information to other community members. One asked, "How do you really make the story come alive?" Repeatedly, the research team heard about the need to augment the data with reports of personal experiences. One strategy for augmenting data was to get local media to run stories of successes. Many partnership members were substantially interested in this and some were quite good at it. Positive media reports were thought to fuel the ongoing effort required to sustain outcomes.

One of the partnerships benefited from the acquisition of the local newspaper by a more community-oriented owner.

Survey of the Media

The Vermont Research Partnership, composed of the education and human service agencies as well as the University of Vermont, surveyed state media and talked with media leaders. They concluded that both the agencies and media each feel that the other fails to understand their role in the community. From the media's point of view, data on public service paid for by tax money is public information and communities need to understand that it is the public's right to have it reported to them in the paper. The agencies felt that the media care only about negative stories.

Things that would improve the working relationship between the media and agencies included

• Agency and community leaders need to better understand the media's context and working habits. "News basics" needs to be part of their professional development. (Including understanding that most of the information journalists get they have to discard and they work in this torrent of information on very tight deadlines.) As part of this, agencies and communities need to better understand how to write a good press release that will catch the reporter's attention and explain things in a clear and compelling way.
• Communities and agencies need to develop thoughtful communications plans that will relay to the public what they are doing.
• Communities and agencies need to build good relationships with the members of the media who cover the beats of interest.

Based on these findings, the Research partnership is developing an action research project for which they will invite teams from five parts of the state to come to monthly workshops with media and public relations experts focused on positive youth development and working with the press.

Panel Discussion

A panel composed of Michael Bennett, Mary Nelson, and Arlene Andrews followed Jennifer Jewiss. Ada Skyles introduced the panel.

Michael Bennett

Context

Michael Bennett began by identifying three experiences that shape his comments. First, he noted that Agriculture Secretary Dan Glickman had released a report that indicated that a family with a child born in 1999 could expect to spend about $160,000 raising that child to age 18 years (with inflation factored in, the total rises to $237,000). Bennett said that these figures indicate to him that more children will have inadequate resources for their growth and development.

Second, he noted that the preceding day, while he attended a Council on Foundations meeting in Los Angeles, he went to a breakfast in which a panel of celebrities talked about how they can be useful for particular causes. Rob Reiner impressed Bennett in his description of how he had dedicated himself to the Clinton-Gore goal of having children ready to learn by the time they get to school and how he was trying to think comprehensively about the components of school readiness--such as health, families, and communities.

Third, Bennett, a graduate of Kent State University, pointed out that 30 years ago to the day four Kent State students ⁽⁵⁾ were shot to death, and others injured, by the Ohio National Guard. Bennett said that the demonstration at which the students were killed was: "about people taking risks because they thought they had the ability to change, about often unmovable forces of government, and about unintended consequences that impact our lives."

Definitions of Community

Bennett offered three definitions of community as it related to the work of supporting children and families. These were

• A geographic definition, something that, on the surface, seems clear but has the potential to be more complex as there are neighborhoods within communities.
• A definition by ascribed characteristics, such as youth, older people, the poor, female-headed households, etc.
• A definition based on a state of being, such as communities in transition, in which immigration is substantial and there is a visible influx of a new population.

Communities and Indicators

Bennett had four points to make related to communities and indicators.

• Communities and community-based organizations have undergone development and have become very sophisticated. But there are some whose public relations image is better than their actual work. It is important to separate fame from accomplishment. (He contrasted having "30 years of experience" with having "one year of experience, thirty times.")
• A lot of communities are not aware of the existence of state data that they could use. They are sometimes more familiar with federal data, such as Census data, but not with possibly more relevant state datasets.
• A number of communities are undergoing or have undergone community-planning processes and have been funded by larger foundations to look at a wide variety of issues. Involvement with such processes could be rewarding for states.
• A number of relationships can be developed that prove useful, such as university-community outreach partnerships, although the barriers to such relationships are formidable.

Bennett said that communities can be very important in helping data gatherers interpret and use data. Meanings may not be obvious to outsiders.

Mary Nelson

Mary Nelson began her portion of the discussion by describing Bethel New Life, a faith-based organization involved with approximately 10,000 individuals who live on Chicago's West Side. Bethel New Life is a real estate development organization (including commercial and industrial development) but, in addition, it is focused on strengthening and supporting families in a community context. That latter part of their mission includes involvement with the Chicago Case Management and WIC (Women, Infants, and Children) programs, programs for the elderly, welfare-to-work programs, child development programs, daycare homes, and other activities.

Bethel New Life has been involved in a variety of programs, including a state infant mortality reduction program in the early 90s that required the organization to keep track of 100 process objectives. They are currently in the fourth year of a five-year demonstration project involving the American Hospital Association called the Outcomes Tool Kit. In most of the other sites in which this project is under way, it is housed in big institutions, hospitals, or government departments.

Bethel's Own Indicators

Nelson said that Bethel New Life had developed its own set of indicators in order to assess whether things are getting better for families in their area of activities. They developed their indicators using this definition of a healthy, sustainable community. A healthy, sustainable community has:

• Economic security
• Ecological integrity
• High quality of life for all
• Public participation in decision making

They are committed to tracking their measures over five years. They have analyzed their findings, and produced easy-to-understand reports that are designed to be useful to those who live in the community. Bethel New Life is very interested in developing approaches focused on outcomes. Nelson notes that the thinking of many community-based organizations is focused on outputs--that is, measures of services provided--and that changing this thinking to an outcome-oriented approach is challenging.

Nelson had four points she wanted to make with the meeting participants. First, if state agencies and other players want community-based agencies to be involved, there must be funds to support this work beyond the per-child service funds. Second, communities must be real partners. They must lead change, not follow the state's agenda. Third, the work must start with assets, not needs, with communities identifying their own strengths and assets. She also said that efforts to work with communities should start not with needs, but with assets. Efforts that focus first on outcomes, then on capacities, help avoid the trap of becoming overwhelmed by needs. Finally, she said that the training process must work both ways and there must be an appreciation of what communities have to share.

Moving to data, Nelson said that the data must be sliced in ways easily accessible. It should be organized by community area, not by zip code, for example. And it should be put into public libraries where it is easily available to communities.

Endnote

5. Allison Krause, Jeffrey Miller, Sandra Scheuer, and William Schroeder.

Session 4: Institutionalizing and Sustaining the Use of Indicators (May 5, 2000)

This session took place on the morning of May 5. Presenters were David Ayer of the Maryland Governor's Office for Children, Youth, & Families (OCYF) and Jim Witherspoon of the Maryland Department of Human Resources. Before the session began, one member of the audience asked that the speakers discuss carrying indicators programs through changes of administration. A second audience member asked that Maryland describe the structure and staffing of their indicators project.

David Ayer

Ayer said that the Maryland presentation on institutionalizing and sustaining the use of indicators would focus on two topics, his talk on communication between the state and communities and Witherspoon's on intergovernmental efforts to institutionalize results and indicators.

Maryland's systems reform effort, and subsequent indicators program, began in the mid-1980s within the human services system and in concert with an Annie E. Casey Foundation demonstration project. Maryland sought to assess how it might better serve at-risk children and families on a variety of health and social needs--then apportioned among a number of agencies, including the Departments of Human Resources, Health and Mental Hygiene, Education, Juvenile Justice, Aging, and Housing and Community Development, and the Office of Individuals with Disabilities--and to create a coordinated system, locally based, that would provide supportive services, not just address crises.

The Subcabinet for Children, Youth, and Families & the Local Planning Entities

Maryland Established a Subcabinet for Children, Youth, and Families to facilitate comprehensive effective and efficient integration of the service delivery system. It also established local planning entities (eventually called Local Management Boards or LBMs) to plan human service delivery in each jurisdiction and provide a central place for local decision making. The LMBs included community members, nonprofits, local elected officials, and human service providers, not just local public agency representatives.

The Task Force on Children, Youth, & Families

During the mid- to late 1990s there was unhappiness in some sectors with system reform and pressure to kill it. Governor Paris Glendening, whose administration had taken over from Governor Schaefer's created a Task Force on Children, Youth, and Families Systems Reform chaired by Lt. Governor Kathleen Kennedy Townsend. One result of the Task Force's work was an invigoration of Systems Reform effort with

• A focus on results and indicators
• An expansion of local authority to determine service needs, reaffirming the LMBs
• The creation of five-year community partnership agreements between jurisdictions and the state

A second result was a reaffirmation of the mission of the system reform effort. In 1989, the Subcabinet defined the mission of child and family services this way:

The mission of services to children and families in Maryland is to promote a stable, safe, healthy environment of ALL children and families, thereby increasing self-sufficiency and family preservation. This requires a comprehensive, coordinated interagency approach providing a continuum of care that is family and child-oriented and emphasizes prevention, early intervention, and community-based services. Priority shall be given to children and families most at risk.

This mission was reaffirmed in the 1996 final report of the Task Force.

Nearly $100 million in state funding was directed by the Subcabinet to the LMBs and community partnership activities (such as Healthy Families and after-school programs).

How the Systems Reform Effort has Driven Maryland to Results and Indicators

The Task Force identified eight (originally nine) results areas. These were:

• Babies born healthy
• Healthy children
• Children enter school ready to learn
• Children successful in school
• Children complete school
• Children safe in their families and communities
• Stable and economically self-sufficient families
• Communities supporting family life

A Results Workgroup was established to operationalize these results. The Workgroup included LMB directors and staff, county social services staff members, juvenile justice staff members, public health officials, district school superintendents, state agency officials, and advocacy group representatives. Expert advice in specific results areas was provided by individual members of the Workgroup and by other experts. The Workgroup's meetings were facilitated by staff of the Center for Assessment and Policy Development (CAPD), a not-for-profit organization located in the Philadelphia area. The full Workgroup met ten times over the course of ten months, between April 1997 and January 1998.

Mark Friedman of the Fiscal Policy Studies Institute developed, in conjunction with the Finance Project and the Center for Study of Social Policy, some very practical guidelines for those developing results-based accountability systems. Maryland found Friedman's guidelines on selecting indicators to help operationalize the results extremely useful. The guidelines included these principles

• The indicator had a known relationship to the dimension of child, family, or community well-being identified in the results area
• The indicator was reasonably well-measured (that is, it applied to all or most of the relevant population and was collected in ways that supported data reliability and validity)
• Data on the indicator were readily available at the present time from public sources.
• Data on the indicator were available at the local jurisdiction level.

Public Engagement

With the help of CAPD, and after a number of sessions, just over 25 indicators were selected. The Work Group's final report, which included recommended results and indicators, was issued. The report was widely distributed and posted on a web site for comment in May 1998. To obtain comments, the report was widely distributed, placed in libraries, and posted on a web site. In addition, a series of local roundtables, hosted by Local Management Boards, were held to enable public review and feedback on the recommended results and indicators. A brochure was developed and distributed to solicit community input and advertise roundtables. A total of 3,850 brochures were mailed to groups and organizations around the state. Public Engagement: Community Input Packets that were prepared for forums held:

• A copy of the report
• A sample regional meeting agenda
• Key presentation points to ensure consistency in reporting
• A framework for small group discussion appropriate to each of the results areas
• A standard format for reporting feedback

Twelve roundtable discussions were held with participation from a total of 300 individuals. A broad diversity of individuals participated, including elected officials, librarians, representatives from community organizations, school personnel, health care representatives, LMB members, and others.

LMB directors and participants appreciated having an independent facilitator to guide the discussion and help solicit local input. The material distributed and format of the roundtables was viewed as helpful, particularly the explanation provided for each result and the time dedicated to discussion. Following the roundtables, a statewide public hearing was held in Annapolis, to offer a forum for citizens, organizations, and agencies who could not attend a local roundtable to testify before a panel of partnership members. Participants were asked to present their concerns, as well as submit written testimony.

Summary of Roundtable/Public Hearing

Six result areas remained virtually intact

• Babies born healthy
• Healthy children
• Children successful in school
• Children completing school
• Children safe in their families and communities
• Stable and economically self-sufficient families
• Added Single Parent Households

Three result areas underwent changes

• Children enter school ready to learn-preliminary indicators
• Healthy adults-dropped
• Communities which support family life-Left for LMBs to develop

The indicators that go with these results include the following.

Babies born healthy. Infant mortality, low birthweight, births to adolescents.

Healthy children. Immunizations, injuries, deaths, substance abuse.

Children enter school ready to learn. Children entering kindergarten with preschool experience, children enrolled in early intervention programs (e.g. Head Start, Infants & Toddlers Program), Low-income children in Head Start or prekindergarten programs.

Children successful in school. Absence from school, academic performance, demonstrated basic skills.

Children completing school. Dropout rate, high school program completion, high school diploma, graduation/school completion of children with serious emotional disturbances

Children safe in their homes and communities. Abuse or neglect, deaths due to injury, juvenile violent offense arrests, juvenile serious non-violent offense arrests, domestic violence.

Stable and economically self-sufficient families. Child poverty, out-of-home placements, single-parent households, Permanent placements, homeless adults and children.

Communities supporting family life. Each LMB to develop its own capacity measures.

Current Organization of Systems Reform

Today, Maryland's systems reform initiative is implemented on the local level through LMBs. Planning is based on local needs researched through a needs assessment process. LMBs are charged with developing a local strategic plan for services. Directors and chairs attended the state's Managing for Results training focused on the state's strategic planning model. The LMBs utilize Maryland's report Results for Child Well-Being to set results for their programs. All 24 jurisdictions have signed contracts with the Subcabinet for the past 3 fiscal years. These contracts fund family preservation and return diversion services, as well as other prevention programs such as disruptive youth, youth suicide prevention, teen pregnancy, prevention, and child abuse and neglect prevention. These contracts are results-based and establish performance measures for each program funded by the LMB. Many and eventually all LMBs will have Community Partnership agreements, multi-year agreements in which jurisdictions extend their reach into result areas in which they need to focus their resources.

LMBs are not service providers. They contract for services with both public and private providers. They receive their funding for these contracts through their individual grant contracts with the Subcabinet. LMB staff members are trained in managing contracts for results. They monitor the service contracts that they fund, and the contracts are developed with set performance measures to hold the vendor accountable. The state's interagency monitoring team conducts evaluations of the contracts, and in conjunction with the LMB establishes a technical assistance plan to ensure that any weaknesses are addressed.

In order to help local jurisdictions, public/private agencies seeking contracts/grants, and the general public with the task of organizing results and indicators, OCYF has taken on the effort to maintain a web page that provides a detailed overview of the selected results and indicators, state level data/graphs, data downloads containing jurisdictional breakdowns, and links to other state and national web sites. The next step for enhancing this site, based on user feedback since it opened in January, is to provide more readily accessible data and graphs at the jurisdictional level.

Simultaneously, the state has been engaged in its efforts to establish a results-based system of accountability at the state level-across all state agencies, not just human services, using the Managing for Results framework, which will be discussed in the next segment of this session.

Reactant

Christine Johnson sketched the similarities between the situation in Maryland and that in Florida, noting in part that many Florida agencies are moving toward a local focus, but does not have a children's subcabinet.

James Witherspoon

James Witherspoon took up the second part of the Maryland presentation, intergovernmental efforts to institutionalize results and Indicators. His presentation was anchored by a series of slides. These follow.

 

Closing Session (May 5, 2000)

Harold Richman of Chapin Hall thanked all who attended, introduced members of the Chapin Hall support staff, and then introduced Martha Moorehouse and Ann Segal to provide concluding remarks from ASPE, the primary project funders.

Martha Moorehouse

We have been thinking about the project and where we go from here. We look forward to hearing your plans about where you're going next. I think that the charge to you is really to focus on institutionalizing. We're less concerned about your thinking about how to institutionalize your existing project structure, but really focus on what you need to do to move the work forward for a long period of time. We have been thinking about ways that we could be a part of helping that. One idea we are thinking about--and we'll be glad to have comment on this as we go--is the possibility of having an out-year meeting which would be developed with Chapin Hall. It would not be just show-and-tell. It would be a working meeting to talk clearly about sustainability and institutionalization issues.

We'd also thought that we just now are really engaged around the issues of community work and connections from state to community that have been a thread throughout. Often the sustainability is really tied into the extent to which this is meaningfully related to community-level work and development. (Summary of the out-year meeting held in May, 2001, is available on this website.) So this is an appropriate focus for an out-year meeting. So that is where our thinking is and it has some way to develop.

We are beginning to summarize across the state projects some of the areas of work that have been a concentration. We're interested in what states have been producing. We know that there have been leadership activities. We know that there have been public engagement activities. We know there has been topical work on particular areas of indicators. There have been report releases and we're interested in trying to capture those and to give all participating states more information on what is coming out of this, and show other audiences what has been accomplished.

Also, we always wanted to reach the larger group of states who are engaged in this work, or who should be engaged in this kind of work. So we've got to keep pushing that along and get information out. Chapin Hall will be developing additional reports.

Ann Segal

ASPE didn't expect, with the amount of money given to states, that there would be huge leaps. But it has been really satisfying to watch how far states have been able to go, given the limited resources out there. A lot of that comes from states sharing with each other, and we knew that was going to be the most valuable part of this project in the beginning.

We hope that, even beyond the six remaining months of the project that states will continue to share information with each other. This project has helped foster new alliances among child well-being projects.

Funds for this effort are among those that will be used to better understand welfare outcomes. We've reported to congress as we've gone along that this is one of the projects for which we've used money under that category, to develop better data at the state level. One of the topic areas we need to continue to work on is how we understand children's well being during a time of welfare policy change.