 |
Wednesday, May 30, 2001
Harold Richman and Mairead Reidy of Chapin Hall welcomed the participants and introduced Martha Moorehouse of the U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation.
Moorehouse began by explaining why it was important to meet, even though the grant period has ended. She said in part:
We thought a lot about whether it made sense to do a meeting after the project was over. We went into the project believing this was about working with states where they were and helping them move along, individually and together, in work with child indicators. Fundamentally, this project is part of government working both at the national level and at the state level in sustaining work on indicators. We decided that if this is about sustaining work on indicators, then we would like to see you after the grants, with the idea that this was part of your sustained work.
Moorehouse said that through the input of the states, the resource people attending this meeting, and the staff at Chapin Hall, the meeting was developed as an effort not just to show off what had been accomplished, but to move the work forward. She and the meeting planners had assessed the remaining issues and work that was still needed, and identified the assistance and discussion that could be helpful. Moorehouse noted that Ann Segal, formerly of ASPE and now of the Packard Foundation, had been a big part of the effort and had joined the meeting. She also said that ASPE will continue, through its own and interagency activities, to make available a variety of federal statistics on child well-being.
Moorehouse said that the Indicators project came out of work that ASPE launched as welfare reform was occurring and was part of a larger policy interest in examining effects of welfare reform. The effort began by working with states that were conducting specific welfare evaluations that included outcomes for children. Some of those early studies have not been published. That work reinforced ASPE's feeling that the opportunities to conduct randomized experiments to illuminate policy issues involving children are few. This project was aimed at working with states to help them develop tools to monitor child well-being over time, and to do this work in partnership with state government and with a strong focus on building state capacity.
Reidy said the aim of the meeting was to help sustain the work by providing an opportunity to discuss successes and new challenges and also to have a chance to reflect on the future use of indicators. She thanked those who had helped shape the agenda and welcomed invited guests.
The first speaker was Cornelius D. Hogan, Senior Consultant with the Annie E. Casey Foundation of Baltimore and former Secretary of the Vermont Agency of Human Services. He was introduced by Harold Richman of Chapin Hall.
Hogan was followed by Gwendoline Angelet of Delaware, Christine Johnson of Florida, and Michael Lahti of Maine. Most sessions from these meetings are summarized. In this case and a few others, we offer a transcript of remarks by the speakers.
When we think of indicators and outcomes, we have to think about neighborhoods. We have to remember we are working on the behalf of neighborhoods and communities. That is what this work is all about. It is at the neighborhood level, at the community level that indicators have a context. And indicators have to have a context. An indicator without a context is like a great meal without ambiance. It is like a politician without an audience or a sailboat without wind. It is the ambiance, the audience, the wind that brings them to life. Context brings an indicator to life.
Indicators are not worth much unless they are put in a strong context. Another word for that context is a common purpose. Without a strong articulated common purpose, indicators don't have resonance. In fact, it is very hard to talk about indicators without talking about the common purpose. One way to guarantee a powerful context is to formulate indicators that describe cross-programmatic, cross-organizational, cross-sector outcomes. Similarly, contexts that are cross-programmatic, cross-organizational, cross-sector will bring indicators to life in a powerful way.
When we discuss indicators it is important that we not become involved in the discussion of specific programs and indicators for the success of that program. The idea of measuring accountability for a specific program mires us in a discussion of the details of those specific programs--often this is a negative discussion of specific programs--and it doesn't allow us to discuss indicators at a higher level. It also prevents us from placing those indicators in context.
To have a broad impact, the outcomes must resonate. There are a number of ways to obtain outcome resonance:
Here are some examples of indicators and outcomes written with the above suggestions in mind:
These outcomes are stated in positive terms. If the outcomes are positive statements, they motivate. Whenever you can restate a negative outcome in positive terms, you have unlocked another level of motivation. Here are some examples of powerful outcome statements that have made a difference:
Win the war. One of the most powerful outcome statements this country has ever been touched by was "win the war." I am old enough to remember my mom collecting tinfoil during the war, collecting her nylons. Everybody knew they had something to contribute. They weren't told what to do.
Here is a case study in the Trondheim region of Norway, which has about 120,000 people, about half the population of Vermont. Their bureaucracies are fascinating. They are much more vertical than the ones here, there are more steps from top to bottom. These bureaucracies are also much more specialized. They will take our child welfare function and break that up into three bureaucracies. By the time you get to the top of their bureaucracies you really can't see what the other guys are doing.
What is the common purpose they could organize themselves around, no matter what bureaucracy they were part of, no matter what organization they belong to? This is the language they came up with: "All children have a human relationship they can depend on." Think about the power of this indicator. The stronger, more powerful, more understandable the common purpose is, the more the context brings indicators to life.
Powerful outcomes and their indicators:
Preserve the culture. This is a raging issue for native Hawaiians and what role they are going to play in the future. Preserve the culture. That language meant something to them. They understand it. They can measure it. It is a very powerful idea.
As we listen to the reactions in the different states we ought to have a red flag that goes up every time we inadvertently bring up an indicator that relates to a specific program. If you really want to keep this powerful, make sure you talk about cross-sector indicators. Indicators that touch more than one program, more than one organizational unit, more than one personality, more than one leader, have much more resonance.
Another thing I've seen states do that you should watch out for is put together indicators for only two or three years. You are laying a trap for yourself if you put together indicators for only two or three years. You really have to collect indicator data for a decade or so. A few years of indicators can get connected to a single administration. You really should be looking at this stuff over the course of a decade to really cross sectors and work out all of the political things. Otherwise, it can really grab you by the neck.
If indicators are used as part of a positive challenge, they communicate volumes. Example: What can you do in your organization to improve teen pregnancy rates? Being able to ask a question and get answers is part of what makes the indicator process an actual process. You have to be able to convert indicators into a personal and interpersonal challenge. When I was Secretary of Vermont's Agency of Human Services I worked with 12 different departments. I always asked the commissioner of each department the same evaluation question every year: What did you do, or can you do, to improve the well-being of the people we serve as described by our agreed upon outcomes? You would be amazed, when you personalize it that way, how the indicators take on importance and become key to the work we all do.
When you have your indicator data, map it out. Part of the indicator challenge is not just talking about indicators, but mapping your data out, putting it into graphs, or some other format, that can easily be used by the press to get word of your program out. It is essential that you find a format that makes it easy for the press to "steal" and place on the news.
Don't focus on just one indicator. There are indicator herds. In conceptualizing indicators, there are always leading indicators which the other indicators follow. If you can find one or two that are going in the right direction, you can be guaranteed you will have 4 or 5 going in the right direction. It is all connected. For example, examining lowering infant mortality leads to considering lowering smoking rates for pregnant women. That in turn leads to thinking about higher education levels, because women who drop out of school are more likely to become pregnant which leads to lower teen pregnancy rates. Lowering teen pregnancy rates leads us to lower child abuse rates. And so it goes.
Everything is connected. Anywhere leads to everywhere. The more you think about your indicators in a connected way, the more you get an integrated view of your work. And the more integrated your view is, the more you can see how it all fits in context, the more your work will resonate with others, and more satisfying and effective your work becomes.
Another way to work with your indicators is to connect changes in indicators with costs. If you do this, you may be able to open up a whole new realm of discussions with a range of people you never thought possible before. You may also be able to bring new arguments to your political process.
If you are able to compute the correlation between indicators and cost you will be able to build relationships with businesses and businesspeople. You will, in a sense, be meeting them on their turf because this is how they think: in profits and loss, benefits and costs. Businesspeople may not know the technical side of teen pregnancies, for example, but they do know that if you have fewer teen pregnancies it costs the state less. Having these numbers available can be very powerful politically. Once again, map these numbers, put them into a graph that states your message simply and strongly. If you describe the movement of your indicators in this way, you have provided a context that will speak to more people and will inspire a sense of pride in your accomplishments.
To survive leadership changes you have to get to the point where your indicators and outcomes are embedded in the culture. One way to do this is to connect the concepts and language of your initiatives to those of other sectors and initiatives. You can, for example, connect with businesspeople in your community by using a modified balance sheet to show the equivalence of positive equity and positive well-being for children, families, and communities. One way to look at indicators is as something akin to a business balance sheet. Our equivalent to the items on a balance sheet is the well-being of our people. The classical balance sheet has been around for a couple thousand years. It can easily be adapted for your purposes. How do the ideas of a balance sheet map across to key indicators? If you have improved indicators, that's a positive cash flow. Improved indicators are short-term assets. Long-term assets include outcomes and indicator structures.
Businesspeople understand balance sheets. They especially appreciate methods of measuring intangible assets. For a business, intangible assets include good will, the capacity and experience of the workforce, customer value, and leadership. Our intangible assets include common purpose, political credibility, community engagement, community assets, and leadership. If you put your indicators and outcomes into this context you will see businesspeople light up. They will trust you and believe you know what you are doing. If you can connect your indicators language to other sectors, you know you have got something that is universal.
I'll end with this fundamental thought about outcomes. The more complicated the organization, the more dangerous the situation, the longer the time line, the larger the area, the more people involved, the more complex the information, the more intense the politics, the more compelling the common purpose must be.
What I'd like to talk to you about for the next few minutes is the practical side of transition and the practice of trying to survive changes in leadership within a state as it relates to social indicators of children's health and well being. In 1998, Delaware published its first Families Count/Kids Count fact books through collaboration with the Kids Count Project administered by the University of Delaware. We have a wonderful relationship with our Kids Count project in Delaware. I think it will be the entity that helps us maintain the momentum as we go forward promoting the use of indicators in Delaware. They have been our strong partner in this effort over the past couple of years. The indicators we use are broad-based indicators that cut across organizational lines, sector lines, geographic lines, that try to paint a complete picture of what is happening with children and families in the context of their communities. What is important to remember here is that we started in 1998.
With the support of the federal grant from U.S. DHHS on using social indicators to promote children's' health and well-being, we broadened the engagement with communities. To this end, we developed a video. We have just completed the video and we are just now distributing it across the state for community groups to use. We have a guide that goes along with the video about ways you can get involved with your community to advocate on behalf of children--one child at a time, or in a group setting. We also have an evaluation form for those people who want to give us feedback about how the video is being received and some action steps they've taken as a result. We'll be able to compile this information and use it to inform our process as we go forward.
These products were done during the administration of our former governor, Governor Thomas Carper. He was a strong believer in what gets measured, gets done. That tone emanated from the top and through our agencies in state government. It resonated with our partner, the University of Delaware, and through our community. He created the Family Services Cabinet Council, which was an important vehicle for bringing together, twice a month, the state agencies concerned with children and family issues.
The Governor personally chaired the council. We had an active agenda and made much progress on children's' issues during those 8 years.
What's Happening Now in Transition from Governor Carper to Governor Minner?
One of the things I am finding out--and I've been through four transitions (I've been in state government a long time)--is that this transition is very different from any that we have gone through. You don't think when you transition from one governor to another from the same party that you are going to go through a complete turnover in the leadership of organizations. We really haven't in the past. But we did this time. That effect is being felt as we try to maintain momentum around those things we thought were working in the prior administration.
Transitions take a long time. They don't happen in just the first 100 days. Because your secretaries get appointed, then they appoint division directors or agency heads, who appoint others, and you have a series of changes, a trickle down effect. So it takes a long time to put it all in place. The learning curve that the new team had takes a while. It's funny, I understood this before, but I understand it now in a very practical way, as we try to maintain the continuity in our work on the indicators.
The new administration is really trying to figure out very basic things like making sure all our computers work, making sure that day-to-day operations are going on. At the same time they are trying to appoint their agency heads. In this case, our Governor is very organized. She had a list of every thing she wanted to get done. She hit the ground running.
Her list of things to get done was very much related to what her campaign platform had been. She didn't hear a lot about social indicators of children's health and well-being on the campaign trail. So that was not at the top of her list of items she wanted to take action on. Then, the 2000 fact book for Families Count/Kids Count came out in January. We invited the Governor to come for the initial press conference.
Some of the indicators in the book were not going in the right direction. Her new team didn't feel comfortable with her coming because they thought she might be put on the spot and they weren't prepared to deal quickly with a response.
We thought we'd done our homework. We met with key advisors before the new team took office. We had the support of new cabinet members. We thought we were doing all the right things. But we came to realize we weren't. We thought what the governor could do is say she was going to continue the work of the cabinet council and give them the charge to work on the indicators that were not going in the right direction and come up with some comprehensive strategies that could address those. However, we pushed a little too hard and as a result, were unable to make much progress.
By the time Kids Count was having its first annual conference in March--and we were actively participating in that--the Governor felt a little more comfortable. She had a little more time to look at the information. Her immediate staff felt more comfortable and that was important. So, she was able to come. She talked about supporting, in concept, the indicators and how she wanted to continue to work around collaboration. She announced that she would appoint an interagency group for bringing the different sectors together to do this.
What Did We Learn From This?
One of the things that really hit home with me was the idea that, if I had it to do over again, I would have produced the video within the first few months of our grant project and gotten it out there. And we would have put the community engagement piece into high gear quickly, really quickly. Because if the governor had heard, while she was campaigning last fall, that citizens were concerned about this indicator or that indicator, it would have resonated with her and her team. I think the other thing that we would have done is spend more time with the governor's team--even during those campaign days--and built those relationships. We would have had an easier time with the top level, so that when we invited the governor to the press conference in January, they would have said, "Oh, it's okay. We know this is what is happening here and we can work through it."
I think the other thing we would have done--and we are trying to do this now--is talk to the legislators about this book. I think when we start to break the data down and make it more personal for individual legislators, personalize it for their individual senatorial or representative districts, then it is going to be more meaningful for them. And they're not going to think about this as a big book with lots of data. They're going to start to think about this in more real terms, in terms of the impact this is going to have on their constituents. So the degree to which we can personalize this information for the legislators, I think that's going to help us reach out.
Another thing we learned from this is that, at the 35,000-foot level, everyone says the indicators are good. But when it comes time to translate the indicators into very practical usage, whether it's advocacy in the community or whether it's policy making and allocation of resources at a state agency, that's a whole different ballgame. I think we have to work to help people make those translations. For example, consider children's readiness for school, and all the indicators that fit into it. We have to help people break this down into a usable factoid, that they can make sense of and really act on.
So where do we go from here? University of Delaware is the entity that is helping us take the important next steps for maintaining the use of the Kids Count, Families Count indicators and the use of the video. We're going to promote the video more. We're going to use these as tools to strengthen our relationships. And one of the things I am very much assured by is that the money that's in the state budget to support Kids Count, which includes production of the Families Count indicators, is still there. Nobody has redlined it. That is a very good start for us because we are really crunching for resources in our state.
The last thing we are going to have to do is continue to have our partners rally around a common purpose. My new boss is a planner and a marketer. What she came up with--which is really helping us--is "Think of the child first." That's become our motto. But I think it is the kind of motto that resonates with a lot of people in terms of looking for that central purpose, that common ground. So, as we go forward, we are going to think of the child first and hopefully that will give us that rallying flag for us to keep things moving forward to survive this change of leadership.
Political context: Changes in power and leadership affect commitment to indicators projects, which by their nature, need to be long-term:
Changes in administration and political party control that affect public policy agendas and priorities. In general, newly elected officials want to "make their own mark." This is particularly when there's a change in political parties. Projects started under a prior administration may not be continued at all or may change dramatically in their form or emphasis. Also, differences emerge in the perception of government's role not only in solving problems, but also in how these problems are measured. In Florida, these issues were manifested as follows:
Term limits: In Florida, over half of state House and Senate members were new this year because of the beginning of term limits. As a result, there is less potential for continuing prior commitments and initiatives. New legislators don't necessarily understand the history of why indicator or other initiatives were developed - or buy into the idea. Also, they bring different ideas about how to set policy and run government.
Organizational Structure and Change
Demographic context: Demographics may play a part in the type of indicator systems developed and how public buy-in is achieved. Florida has an ethnically diverse population and a large percentage of seniors (age 65 and older) relative to the rest of the country. Most indicator systems developed at the state and local level do not focus on specific subpopulations (e.g., children only, African Americans only). They include indicators relevant to these subgroups - but embedded in a broader range of indicators geared toward the general population. Within this framework, subpopulations are addressed through specific indicators (e.g., low birthweight) or breakdowns by race, age, region or other factor.
Local context: Large urban areas have the capacity to start and continue indicator projects on their own resources--without depending on state support. This has certainly been true in Jacksonville, Palm Beach, Orlando, Fort Lauderdale and other metropolitan areas of Florida. In the long-term, these projects may be more stable because of their independence and closeness to the issues of greatest importance at the local level. On the other hand, rural communities with fewer resources, tend to be left out of the picture.
Leadership for indicator projects, at least in Florida, appears to be more stable at the local than the state level - although the same political uncertainties apply. The Jacksonville Community Council, Inc. indicators project, for example, has experienced substantial fluctuations in annual funding depending on support coming from the mayor's office or the chairman of the local chamber of commerce. What has helped to sustain this 15-year initiative through these funding ups and downs is (1) the participation and buy-in at the community level which was strong from the start, (2) the use of indicators to do studies of local issues and problems, (3) the public's trust in the quality of data and (4) the public's continuing expectation of data availability.
Public and institutional reaction: Reaction seems to be generally positive to indicator systems that seek to "generally inform"--without consequences. Problems usually arise when these systems actually get used, for example, to cut funding for programs or to grade schools "A" to "F" on student achievement. When decisions "hit close to home" reactions are likely to be most intense. For example, based on anecdotal evidence, there seems to be a backlash among parents of high-achieving students in public schools in Florida, who are concerned about how their children will be affected by what they perceive to be "high stakes" testing. Likewise, schools, agencies and other institutions can get defensive about what indicators say - particularly if the public or the media use the data to criticize them. Developing buy-in and creating a climate of cooperation can mitigate these effects but are difficult to achieve, particularly when public opinion is divisive or state-local relationships are mistrustful.
Measurement context: The range and availability of data are continuing to increase at the state and local level--a real plus for indicator projects. Data collection itself is improving, as is the technology for analyzing and distributing information to policymakers and other consumers. However, we still haven't mastered the art of interpreting this information meaningfully in all of its complexity. For example, people still tend to focus on individual measures without understanding problems on a broader, more complex level - a difficult task given the quantity of information out there and the interconnections of programs and outcomes. How do we keep it simple enough to understand, yet avoid misleading policymakers and the public into simplistic solutions?
Policymakers typically want to know more than broad outcomes. They want diagnostic information so they know "what to fix." The challenge is communicating this information - including cause-and-effect relationships - in a clear, concise, timely and useful way. Indicator systems are useful signaling and analysis tools, but at least for policymakers, there is demand for broader synthesis of information from a variety of sources - indicators, research studies, surveys, etc. - to guide decision-making on a host of issues within a relatively short time frame, such as a legislative session. Policymakers are certainly not the only major users of indicator data - but they do have an important influence on state and local capacity to solve problems and improve the well-being of the population.
Our work in Maine has resulted in a project called Maine Marks. Our first publication came out in February. I want to give you a little context and focus on what we have incorporated into the process. Because we have not experienced a change in governorship during the period of this process, we still have three more years with the current governor, who is a big supporter of this. I want to talk about how we've tried to create a structure for this to continue, regardless of what happens.
This has to be bigger than any one person or program. I often think what we are doing is telling a story. I keep thinking about people sitting around a fire at the gates of their community.
You walk up to the first ones and they tell you what is happening in the community. The stories we want to hear become important, not necessarily who the king is.
The first publication for Maine Marks project came out in February 2001. It took us about 2 years to develop. This grant was a huge influence on that--as well as Kids Count--they are a very significant partner.
What happened in Maine is that a governor's Children's Cabinet was established five years ago, not in statute but through a Danforth Foundation grant and the vision of the current governor to bring these agencies together. The group established a set of 12 outcome statements for kids, families, and communities in Maine.
We were given the outcome statements right from the beginning. Our tack was to look at indicators that could relate with those outcome statements. We ended up with 80 indicators, which is a big number. We had a lot of conversations about that number. I think back to a meeting here where David Murphey was talking about not being too afraid of big numbers because then a lot of people can find a place to sit.
I think we have a lot of indicators that are not well defined. But people felt strongly when we were creating the indicators that all of the indicators were important, so we decided to get them down on paper from the beginning.
As for a structure and surviving leadership changes, we were given these outcomes from the beginning. These outcome statements themselves were not well-defined. One of the other things we wanted, and got, was to have at least half of these indicators to be promotional and/or strength-based. That also ended up in a lot of work. We ended up using a lot of material from our Search Institute and other places. We felt strongly that we wanted a comprehensive picture. Again we wanted to take a look at all that was happening with kids and families.
Right now, specific to the work of the project, we are trying to look at data collection for next year, for further definitions or indicators. And we'd love to get county-level data, because right now we just have state-level data for a number of indicators, especially newer ones.
In terms of surviving leadership changes, some of the things we have been intentional about, this connection to the Children's Cabinet is one of them, and being broad in scope in terms of the number of indicators we are looking at. Again, there is something in there for everybody.
The Children's Cabinet is now established in statute. Within the Children's Cabinet is a council that has representatives from the legislature, the judicial branch, and other key stakeholders and policymakers within the state government. So these indicators have become their indicators. Hopefully that will be helpful over time.
We did look at outcome statements that were categorical in nature. We have a few indicators, that when you look at them, are programmatic. I don't know how that happened, but it did.
I think another thing we tried to do--and maintained well--is partnerships. Things couldn't have happened without Kids Count, and partnerships with the Maine Development Foundation were critical, because they had already established indicator work that people used. So our book was formatted to look exactly like their book. That was very intentional. We worked at the Maine School of Science and Mathematics. They helped us at the very beginning to look at these, to develop the web pages, and the connection to the university has been helpful as well.
A big task which has already been mentioned, and which we hope to learn more about, is driving this work down and across systems.
Becky Boober is here; she is amazing for region 3. They have already started within their Children's Cabinet to connect Maine Works to things like training and data usage. They'll use this as content for training and for use in grantmaking. So when people come to them for dollars, they are asking them to connect their thinking to these indicators and program development. As an evaluation work, we are trying to tie things back to this.
The other thing that snuck in and out of this process--and I was surprised it didn't happen more often--was that there were instances when we had to think of how this connected to performance-based budgeting. There are two performance-based government initiatives that had already been established in statute. So those questions got asked and we put together little charts that showed how things were lined up. And then it kind of went away, which was a good thing for me. But it will be back. I think you should be prepared for that, for strategic plans. I think in the future it is important that we remember that this is an iterative process. And I think when we talk about this as actual products that that is certainly not true.
I remember hearing a psychologist or psychiatrist talking about autism on the radio, saying how that problem used to be understood, and how it is understood today. Certainly our language about children is going to change. The other thing is that for me, it always bugs me when I flip on CNN and see the tickertape flipping across the bottom of the channel, and how we think of economic indicators in this country. Yet, we don't have data on teen suicides in Maine. The most current data we have is 1997, because we can't get schools to deal with this whole inundation they have, filling out forms and getting kids to fill out surveys. Right now, it is such a struggle to get people to pay attention to social indicators as important, if not more important, than economic indicators.
I was thinking in closing that is it very helpful to think of this in metaphors. For me, I think this information and these systems are like a painter's palette. Leaders mix paints to create their picture. But that palette and that paint is there forever. We want to make sure that the paint is of the highest quality. And look at that as a metaphor for our work.
The system has to be bigger than whoever that artist is.
This session was coordinated by Ada Skyles of Chapin Hall. The presenters were David Murphey, Senior Policy Analyst of the Vermont Agency of Human Services; Cherie Hammond, Coordinator of the Success by Six Council in Lamoille County, Vermont; Scott Johnson, Coordinator of People in Partnership, also in Lamoille County; and Larry Pasti, Community Program Specialist in the New York State Office of Children and Family Services.
The session structure: Murphey raised and commented on key points which were then addressed by the other panelists. He began by saying that creating a climate in which communities can take advantage of indicators requires activity both from the top-down and from the bottom-up. With the publication of Vermont's first Kids Count book, Vermont communities asked for community-level data, feeling that it would be more useful than the county-level data presented in the volume. The state wanted to respond to this because it sees communities as the locus of change.
Murphey presented the following key points he and the panel hoped to cover [See also full paper]:
Murphey said that "getting local, broad-based buy-in" means that people need to understand that outcomes are about having a common purpose, that they are bigger than any single agency. And they need to understand what indicators are--specific, measurable ways of understanding the progress that is being made toward those outcomes. Having that conceptual framework in place is critical, but within that framework there is flexibility. Having that framework in place is more important than the adoption of a particular indicator. For Vermont's part, it has adopted a list of nine specific outcomes. But it encourages communities to go beyond that list to develop outcomes and create and monitor indicators of particular use locally. In Vermont, this has happened. Some communities have taken on all nine outcomes, some have developed additional outcomes, and some have chosen a smaller number of outcomes on which to focus.
Communities are experts, Murphey pointed out, on many topics--an expertise that the state cannot duplicate. His second overhead pointed out some areas in which communities possess expertise:
What Are Communities Experts On?
Murphey asked the panel to comment on getting local, board-based buy-in. Johnson stressed the importance of creating an army of advocates engaged in a variety of tasks working to forward the common purpose. Hammond said that with the community of those working on early childhood concerns in Vermont, there has been advocacy statewide on a broad-based agenda of early childhood issues. Hammond also told how she and her organization had worked with legislative candidates before the election to raise their concerns regarding early childhood issues, and continued that relationship later, as the state legislature took up issues relevant to young children.
Pasti sketched an integrated county planning demonstration project begun two years ago in selected New York counties. The project had two objectives. The first objective was to help counties be more comprehensive in their planning by bringing together two separate planning projects (one from the local human services and the other from the county youth bureau). The second objective was the way to change how counties plan, to move away from planning based on deficits and services. Among the goals of this effort was to enhance local control of the process, tap into grassroots resources, to broaden the focus of planning, to work the concepts of the human development continuum--such as of health and wellness--into the process, and to focus on outcomes.
As part of this effort, the state required counties to come up with a vision of what they were to accomplish with youth, but did not mandate use of the New York State Touchstones model, even though that had been agreed on by the state's agency commissioners. Touchstones was merely offered to the counties as a model. Most of the counties in the process did choose to adopt an outcomes framework for organizing planning and those that did not choose outcomes directly modified outcomes for their own local conditions. This is an example of how New York is trying to use county government to promote the adoption of an outcomes approach and to get a broad stakeholder involvement.
Murphey said that the comments of the panel helped underscore the importance of the second point from the first overhead, to encourage outcomes-based collaborations and, as a corollary, avoid turf issues. Vermont has 12 regional partnerships. Johnson is the coordinator of one of those partnerships. Each of those partnerships is the keeper of the flame for outcomes within its region and each partnership, in turn, works with other partnerships with many issues and foci. Working at a state level, it is important to encourage those kinds of collaborations, and to help the situation remain flexible in order to avoid turf problems.
Johnson named some of the partnerships with which he is involved in his region and sketched some of the ways in which they interact. Pasti commented on collaborations at the county level. He finds value in the flexibility that New York has allowed counties by not mandating participation. Hammond cited the involvement of parents in their collaborations as a strength of their organization and pointed out some of the ways they encourage parents to participate--including paying stipends to parents and providing dinner and childcare at evening meetings. Murphey echoed the high value placed on citizen engagement in Vermont.
Murphey said that indicators provide a mirror in examining society. The way Vermont holds up a mirror to a community is through the use of community profiles. Now in their sixth years, the community profiles have taken hold and are enjoying increasing use. Interest is enhanced in them by the efforts of the state to frame the book within the context of each partnership area.
Hammond said that the Success by Six project must submit an annual plan for meeting the state's outcomes every year in order to be funded. This process supports and helps shape the work of the organizations in the partnership in a variety of ways.
Murphey said that it is not enough to publish the data. It is also critical that communities review rationally and engage with the data and it is up to the state to promote this engagement, review, and then planning. A first step is helping communities develop a comfort level with data.
Murphey uses this slide in working with community groups:
Why Use Data?
We already know what the problems are!
A member of the audience said that she was struggling with the problem of the language in which the findings are expressed interfere with the data's relevance. Hammond responded that some audiences were going to zero in on particular aspects of the indicator data while others will take a broader view. She illustrated her point by describing comparisons that can be made between low-birthweight babies and smoking by pregnant women by economic group. A comparison of this type can help legislators shape programs to reach pregnant smokers.
Using the slide below as an illustration, Murphey discussed the importance of using multiple sources of information.
Do We Have Multiple Sources of Information?
It's desirable not to rely on a single piece of information, but to have both multiple sources of information for an indicator and to use multiple indicators. Among these multiple sources, it is important to obtain the view of the community affected as to what the information that makes up the indicators mean. There was general agreement among members of the panel about the value of multiple sources of data. Murphey said that in Vermont state government tries to model the data use strategies it believes communities could find beneficial in using these data. The state Team for Children, Families, and Individuals meets once a month. One of the outcomes is the topic of one of their meetings. Every year, out of that process, comes a publication that addresses each outcome and highlights some "headline" outcomes, up to three heartening indicators and up to three troubling indicators. The second step to this process was then to identify some action steps.
Murphey said that the important point is not that there is one right answer on which indicators are the three best or worse, but the process of working together to choose some headline indicators that will motivate strategic action steps.
Murphey's next slides showed some of the questions that could be raised in the context of comparisons and asked questions about presenting data:
Compared to what?
Some Things to Consider in PRESENTING Data
Johnson then discussed a regional youth project that involved repackaging community profile data on youth statuses and risk behaviors for particular purposes. Johnson said that their purpose is to engage the local media and to engage individuals with monthly advertisements in the local newspapers, distributed by mail, and sometimes reproduced for school use.
Murphey said that one of the anxieties created by the production of the first community profiles was that programs working in the areas addressed by the profiles did not understand that these profiles presented population data and were not a verdict on their program work.
To counter this, Vermont developed a language to describe these data to programs. One step, borrowing from the United Way, was to distinguish between the community-level outcomes to which program-level outcomes can make a contribution. In this effort, they work to identify the links between milestones--early indicators of success--and the ultimate indicators of success. This is new territory for many of the community folks.
Pasti said that this is a substantial shift in thinking for counties because they are used to measuring program outcomes. Johnson described a program in Vermont that is setting up youth councils that are involved in making grants. One of the ideas behind this is to teach young people about outcome-based planning and logic models.
Murphey moved to an example of the relationship between program-level outcomes and the ultimate population-level outcomes by presenting a slide on child welfare services in northeastern Vermont.
Vermont Department of Social & Rehabilitation Services, Northeast Region
Outcomes for Children in Foster Care
Permanency Indicators:
Resiliency/ Social Skills Indicators:
The state child welfare agency has developed a monthly reporting system so that the caseworkers can report on the status of children's progress toward these two indicators. This is an example of how the outcomes work has been translated into community program work, linking the program and the population-level data. In response to a question later in the session, Murphey noted that these indicators measure qualities of life important to all children, and are of particular importance to children in foster care.
Question. These numbers will sometimes go in very different directions, and will move incrementally. What tells you that you have a problem and it's not just a problem in statistics?
Answer. Murphey said, I think that the point here is to show continuous improvement. These are the things that you want to show progress in.
In response to a question on whether these are the right measures, Murphey explained that although this represented substantial thought, that all indicators lists are provisional and open to improvement. There was discussion about how to decide which measures are important in a local region and speakers on the panel and from the audience suggested strategies for selecting the right indicators in order to get at issues of community concern. There was general agreement that regions and communities have critical roles to play.
In response to how to deal with communities seeking value-laden indicators, such as church attendance, and how to steer such requests toward a research base, Murphey said that they encourage communities to develop their own theories of change, but to also look at what the literature says in an objective way. (With the understanding that research is an ongoing proposition and what we know today is going to be different from what we will know tomorrow.) So, a group that is interested in measuring the effect of church attendance must come up with a logic model through some collaborative, consensus-based project, and then collect data to test it.
Murphey displayed a United Way program outcome model overhead similar to the one below.
A General Model for Community Planning in Getting to Outcomes
|
|
|
|
|
<------------------------------------------------------------------------Evaluation---------------------------------------------------------------------------->
He said that using three program logic models, as displayed in the overhead, was to indicate that it was likely that more than one program contributes to an outcome. Murphey also said that the shortcoming of the United Way logic model, in his view, is that it assumes there is a program. Vermont has been trying to encourage communities, wherever possible, to back up the process even more and base their selection of a program on a theory of change and on a data-driven assessment of the priorities for their communities.
This is an ideal model of how the world might work; it doesn't always work that way. Ideally, a community will have gone through a comprehensive assessment and will have developed some kind of a theory of change based on best practice, based on what the research literature says, based on what is known about a community and its unique characteristics, and so on. This is the grand scheme of how it fits together in terms of evaluation and program.
Once they have prioritized indicators, such as identifying the heartening and troubling indicators, the other half of that is to assign tasks to specific people. They have tried to hold people's feet to the fire, including everyone from commissioners to program staff, to say, in effect, "this is what you can do on this indicator and we expect to hear back from you the next time we address this outcome at the state team meeting a year from now and hear you report on what you have been able to accomplish."
Johnson described how this process played out in the Lamoille Valley Reads program as many groups used multiple strategies to promote literacy. Partners included the schools, Success by Six, and service providers. Media coverage was a part of their strategy.
Murphey added that, to keep all this work from being just talk, there is a point at which assignments need to be made and a reporting schedule need to be made--who is going to do what by when.
Pasti said that one of the difficulties they face in New York is that they have plans coming from primary county agencies in social services and youth work. These agencies typically submit plans for their own work. The state tells the agencies that it wants to know not only what any individual agency is doing, but how the strategy of any one agency includes what other agencies are doing. It helps make each agency accountable by requiring them to put the work of the other agencies in their plan. That's a shift from their traditional, service-driven planning to outcome-driven planning and accounting for the influences of multiple partners. Pasti noted that this illustrated the difference between accountability and responsibility.
Murphey added that one of the messages Vermont government gives is that programs are accountable for achieving outcomes with their clients and they are responsible for the community-level outcomes. There is a difference, but there has to be accountability when there are funds at stake. Plus, you need to be accountable to the people that you serve, as well as to your own staff. So, there is a need for accountability, but it is important to distinguish between accountability and responsibility.
Murphey said that inherent in the devolution bargain is that states and communities are given more flexibility to achieve outcomes. He said that there was a trend in the late 80s and early 90s for "unmanageable" kids in state custody to be placed in very expensive out-of-home and sometimes out-of-state placements. In the Lamoille County area, they experimented with trying to keep the kids closer to home or at home or keep them in-state in exchange for the use of some of the savings that the state child welfare agency would realize to provide additional services. Murphey said that the county did bring down the numbers of unmanageable kids and was allowed to retain half the savings that accrued to the child welfare agency as a result of the lower numbers.
In response to a question about creating projection models of savings that can be secured by averting expensive, out-of-state placement, Pasti said that New York has done a similar reinvestment. One difficulty they encountered was that some counties had already made substantial progress at the time the reinvestment program was implemented. In those counties, financial benefits were more modest than in places where less had been accomplished.
Local Media
Murphey said that an indicators report can be a tool in getting the local media to do a story on a particular indicator. One idea he mentioned was getting a local newspaper to run a story on a selected indicator each month.
Keep Holding Up A Mirror
Murphey cautioned against turning indicators work into a high stakes game or a game of "gotcha" with communities -- tying funding to a community's indicators. Vermont tries to recognize improvement whenever they find it.
Stay in This For the Long Haul
This is really long-term work, Murphey warned. It is a different way of thinking and working. It is much longer than any single administration and it is going to take time both at the community level and at the state level. People at the state level and at the community level are often impatient. They want to see results. They are eager to get the data every year, and, if those data don't show that they have made great progress, it can be demoralizing. It's important to remember that there is not a one-to-one relationship between effort and outcome. But success is seen in efforts in which stay the course.
Conclusions
Each of the panel members had a closing comment. Johnson said that he wanted to thank his state partners -- such as Murphey. He said that there is truly a state and local partnership and that there is a level of respect between the state and the local activities. Pasti echoed the idea that it is critical to stay involved for the long haul and that it is an evolving process. Hammond said that, although there is a lot of information available to her in the community profiles, she also seeks information on her own and she keeps a file on where information can be found.
Home Pages:
Advancing States' Child Indicators Initiatives
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)