This session was coordinated by Ada Skyles of Chapin Hall. The presenters were David Murphey, Senior Policy Analyst of the Vermont Agency of Human Services; Cherie Hammond, Coordinator of the Success by Six Council in Lamoille County, Vermont; Scott Johnson, Coordinator of People in Partnership, also in Lamoille County; and Larry Pasti, Community Program Specialist in the New York State Office of Children and Family Services.
The session structure: Murphey raised and commented on key points which were then addressed by the other panelists. He began by saying that creating a climate in which communities can take advantage of indicators requires activity both from the top-down and from the bottom-up. With the publication of Vermont's first Kids Count book, Vermont communities asked for community-level data, feeling that it would be more useful than the county-level data presented in the volume. The state wanted to respond to this because it sees communities as the locus of change.
Murphey presented the following key points he and the panel hoped to cover [See also full paper]:
Growing an Outcomes-Based Culture with Communities
- Get local, broad-based buy-in on the outcomes and indicators (conceptual level) (with flexibility).
- Encourage outcomes-based collaborations ("set the table"); avoid "turf" issues.
- "Hold up the mirror" of community indicators.
- Promote a rational local review of the indicators, leading to prioritization (requires a comfort-level with data).
- Foster strategies to measure program outcomes as well as community outcomes (e.g., logic models and associated evaluation).
- Identify "turn the curve" strategies with specific who/what/by when action-steps.
- Consider negotiating for greater funding flexibility in exchange for improved outcomes.
- Engage the local media around the outcomes and indicators.
- Keep "holding up the mirror." No "high stakes," but gentle reflection.
- Stay in this for the long haul.
Murphey said that "getting local, broad-based buy-in" means that people need to understand that outcomes are about having a common purpose, that they are bigger than any single agency. And they need to understand what indicators are--specific, measurable ways of understanding the progress that is being made toward those outcomes. Having that conceptual framework in place is critical, but within that framework there is flexibility. Having that framework in place is more important than the adoption of a particular indicator. For Vermont's part, it has adopted a list of nine specific outcomes. But it encourages communities to go beyond that list to develop outcomes and create and monitor indicators of particular use locally. In Vermont, this has happened. Some communities have taken on all nine outcomes, some have developed additional outcomes, and some have chosen a smaller number of outcomes on which to focus.
Communities are experts, Murphey pointed out, on many topics--an expertise that the state cannot duplicate. His second overhead pointed out some areas in which communities possess expertise:
What Are Communities Experts On?
- Their assets
- Their needs
- The "story behind" the data
- Their priorities (value-driven as well as data-driven)
- Their "character" (e.g., aspects of class, race/ethnicity, history, religion, etc.)
- Other local conditions or circumstances that affect:
- Access to and utilization of services and supports
- Risk and protective factors
- "Practice variation" issues
- Community readiness to move ahead
Murphey asked the panel to comment on getting local, board-based buy-in. Johnson stressed the importance of creating an army of advocates engaged in a variety of tasks working to forward the common purpose. Hammond said that with the community of those working on early childhood concerns in Vermont, there has been advocacy statewide on a broad-based agenda of early childhood issues. Hammond also told how she and her organization had worked with legislative candidates before the election to raise their concerns regarding early childhood issues, and continued that relationship later, as the state legislature took up issues relevant to young children.
Pasti sketched an integrated county planning demonstration project begun two years ago in selected New York counties. The project had two objectives. The first objective was to help counties be more comprehensive in their planning by bringing together two separate planning projects (one from the local human services and the other from the county youth bureau). The second objective was the way to change how counties plan, to move away from planning based on deficits and services. Among the goals of this effort was to enhance local control of the process, tap into grassroots resources, to broaden the focus of planning, to work the concepts of the human development continuum--such as of health and wellness--into the process, and to focus on outcomes.
As part of this effort, the state required counties to come up with a vision of what they were to accomplish with youth, but did not mandate use of the New York State Touchstones model, even though that had been agreed on by the state's agency commissioners. Touchstones was merely offered to the counties as a model. Most of the counties in the process did choose to adopt an outcomes framework for organizing planning and those that did not choose outcomes directly modified outcomes for their own local conditions. This is an example of how New York is trying to use county government to promote the adoption of an outcomes approach and to get a broad stakeholder involvement.
Murphey said that the comments of the panel helped underscore the importance of the second point from the first overhead, to encourage outcomes-based collaborations and, as a corollary, avoid turf issues. Vermont has 12 regional partnerships. Johnson is the coordinator of one of those partnerships. Each of those partnerships is the keeper of the flame for outcomes within its region and each partnership, in turn, works with other partnerships with many issues and foci. Working at a state level, it is important to encourage those kinds of collaborations, and to help the situation remain flexible in order to avoid turf problems.
Johnson named some of the partnerships with which he is involved in his region and sketched some of the ways in which they interact. Pasti commented on collaborations at the county level. He finds value in the flexibility that New York has allowed counties by not mandating participation. Hammond cited the involvement of parents in their collaborations as a strength of their organization and pointed out some of the ways they encourage parents to participate--including paying stipends to parents and providing dinner and childcare at evening meetings. Murphey echoed the high value placed on citizen engagement in Vermont.
Murphey said that indicators provide a mirror in examining society. The way Vermont holds up a mirror to a community is through the use of community profiles. Now in their sixth years, the community profiles have taken hold and are enjoying increasing use. Interest is enhanced in them by the efforts of the state to frame the book within the context of each partnership area.
Hammond said that the Success by Six project must submit an annual plan for meeting the state's outcomes every year in order to be funded. This process supports and helps shape the work of the organizations in the partnership in a variety of ways.
Murphey said that it is not enough to publish the data. It is also critical that communities review rationally and engage with the data and it is up to the state to promote this engagement, review, and then planning. A first step is helping communities develop a comfort level with data.
Murphey uses this slide in working with community groups:
Why Use Data?
We already know what the problems are!
- To confirm/revise existing judgments
- To add credibility to your efforts
- To help prioritize efforts
- To provide a baseline
A member of the audience said that she was struggling with the problem of the language in which the findings are expressed interfere with the data's relevance. Hammond responded that some audiences were going to zero in on particular aspects of the indicator data while others will take a broader view. She illustrated her point by describing comparisons that can be made between low-birthweight babies and smoking by pregnant women by economic group. A comparison of this type can help legislators shape programs to reach pregnant smokers.
Using the slide below as an illustration, Murphey discussed the importance of using multiple sources of information.
Do We Have Multiple Sources of Information?
- For an outcome: multiple indicators?
- For an indicator: multiple types/sources of data?
- Qualitative as well as quantitative?
- Multiple "formats"? (e.g., focus groups, interviews, surveys, previously published statistics)
- Multiple time-points? (where we've come from, where we seem to be going)
- From multiple constituencies? (e.g., parents, students, service providers, "regular folks")
It's desirable not to rely on a single piece of information, but to have both multiple sources of information for an indicator and to use multiple indicators. Among these multiple sources, it is important to obtain the view of the community affected as to what the information that makes up the indicators mean. There was general agreement among members of the panel about the value of multiple sources of data. Murphey said that in Vermont state government tries to model the data use strategies it believes communities could find beneficial in using these data. The state Team for Children, Families, and Individuals meets once a month. One of the outcomes is the topic of one of their meetings. Every year, out of that process, comes a publication that addresses each outcome and highlights some "headline" outcomes, up to three heartening indicators and up to three troubling indicators. The second step to this process was then to identify some action steps.
Murphey said that the important point is not that there is one right answer on which indicators are the three best or worse, but the process of working together to choose some headline indicators that will motivate strategic action steps.
Murphey's next slides showed some of the questions that could be raised in the context of comparisons and asked questions about presenting data:
Compared to what?
- On this indicator, how is my community doing:
- Compared to the goals our community aspires to?
- Compared to where we've been (timetrend)?
- Compared to communities like ours?
- Compared to our county's record?
- Compared to the state's record?
Some Things to Consider in PRESENTING Data
- Who is your audience?
- What is your purpose?
- Present both assets (strengths) and needs (gaps, troublesome areas)
- Is the glass half-empty, or half-full?
- What (carefully chosen) comparisons would help reinforce your points?
- Be honest about the limitations inherent in all data.
- Use simple charts to convey the information graphically.
- In terms of communication power, often "less is more."
Johnson then discussed a regional youth project that involved repackaging community profile data on youth statuses and risk behaviors for particular purposes. Johnson said that their purpose is to engage the local media and to engage individuals with monthly advertisements in the local newspapers, distributed by mail, and sometimes reproduced for school use.
Murphey said that one of the anxieties created by the production of the first community profiles was that programs working in the areas addressed by the profiles did not understand that these profiles presented population data and were not a verdict on their program work.
To counter this, Vermont developed a language to describe these data to programs. One step, borrowing from the United Way, was to distinguish between the community-level outcomes to which program-level outcomes can make a contribution. In this effort, they work to identify the links between milestones--early indicators of success--and the ultimate indicators of success. This is new territory for many of the community folks.
Pasti said that this is a substantial shift in thinking for counties because they are used to measuring program outcomes. Johnson described a program in Vermont that is setting up youth councils that are involved in making grants. One of the ideas behind this is to teach young people about outcome-based planning and logic models.
Murphey moved to an example of the relationship between program-level outcomes and the ultimate population-level outcomes by presenting a slide on child welfare services in northeastern Vermont.
Vermont Department of Social & Rehabilitation Services, Northeast Region
Outcomes for Children in Foster Care
- Length of stay
- Average number of unplanned moves
Resiliency/ Social Skills Indicators:
- Percentage attending school or have other educational plan
- Percentage participating in structured after-school activities
- Percentage earning wages
- Percentage volunteering in the community
- Percentage maintaining friendships with age-peers
- Percentage with an adult "connection"
The state child welfare agency has developed a monthly reporting system so that the caseworkers can report on the status of children's progress toward these two indicators. This is an example of how the outcomes work has been translated into community program work, linking the program and the population-level data. In response to a question later in the session, Murphey noted that these indicators measure qualities of life important to all children, and are of particular importance to children in foster care.
Question. These numbers will sometimes go in very different directions, and will move incrementally. What tells you that you have a problem and it's not just a problem in statistics?
Answer. Murphey said, I think that the point here is to show continuous improvement. These are the things that you want to show progress in.
In response to a question on whether these are the right measures, Murphey explained that although this represented substantial thought, that all indicators lists are provisional and open to improvement. There was discussion about how to decide which measures are important in a local region and speakers on the panel and from the audience suggested strategies for selecting the right indicators in order to get at issues of community concern. There was general agreement that regions and communities have critical roles to play.
In response to how to deal with communities seeking value-laden indicators, such as church attendance, and how to steer such requests toward a research base, Murphey said that they encourage communities to develop their own theories of change, but to also look at what the literature says in an objective way. (With the understanding that research is an ongoing proposition and what we know today is going to be different from what we will know tomorrow.) So, a group that is interested in measuring the effect of church attendance must come up with a logic model through some collaborative, consensus-based project, and then collect data to test it.
Murphey displayed a United Way program outcome model overhead similar to the one below.
A General Model for Community Planning in Getting to Outcomes
He said that using three program logic models, as displayed in the overhead, was to indicate that it was likely that more than one program contributes to an outcome. Murphey also said that the shortcoming of the United Way logic model, in his view, is that it assumes there is a program. Vermont has been trying to encourage communities, wherever possible, to back up the process even more and base their selection of a program on a theory of change and on a data-driven assessment of the priorities for their communities.
This is an ideal model of how the world might work; it doesn't always work that way. Ideally, a community will have gone through a comprehensive assessment and will have developed some kind of a theory of change based on best practice, based on what the research literature says, based on what is known about a community and its unique characteristics, and so on. This is the grand scheme of how it fits together in terms of evaluation and program.
Once they have prioritized indicators, such as identifying the heartening and troubling indicators, the other half of that is to assign tasks to specific people. They have tried to hold people's feet to the fire, including everyone from commissioners to program staff, to say, in effect, "this is what you can do on this indicator and we expect to hear back from you the next time we address this outcome at the state team meeting a year from now and hear you report on what you have been able to accomplish."
Johnson described how this process played out in the Lamoille Valley Reads program as many groups used multiple strategies to promote literacy. Partners included the schools, Success by Six, and service providers. Media coverage was a part of their strategy.
Murphey added that, to keep all this work from being just talk, there is a point at which assignments need to be made and a reporting schedule need to be made--who is going to do what by when.
Pasti said that one of the difficulties they face in New York is that they have plans coming from primary county agencies in social services and youth work. These agencies typically submit plans for their own work. The state tells the agencies that it wants to know not only what any individual agency is doing, but how the strategy of any one agency includes what other agencies are doing. It helps make each agency accountable by requiring them to put the work of the other agencies in their plan. That's a shift from their traditional, service-driven planning to outcome-driven planning and accounting for the influences of multiple partners. Pasti noted that this illustrated the difference between accountability and responsibility.
Murphey added that one of the messages Vermont government gives is that programs are accountable for achieving outcomes with their clients and they are responsible for the community-level outcomes. There is a difference, but there has to be accountability when there are funds at stake. Plus, you need to be accountable to the people that you serve, as well as to your own staff. So, there is a need for accountability, but it is important to distinguish between accountability and responsibility.
Murphey said that inherent in the devolution bargain is that states and communities are given more flexibility to achieve outcomes. He said that there was a trend in the late 80s and early 90s for "unmanageable" kids in state custody to be placed in very expensive out-of-home and sometimes out-of-state placements. In the Lamoille County area, they experimented with trying to keep the kids closer to home or at home or keep them in-state in exchange for the use of some of the savings that the state child welfare agency would realize to provide additional services. Murphey said that the county did bring down the numbers of unmanageable kids and was allowed to retain half the savings that accrued to the child welfare agency as a result of the lower numbers.
In response to a question about creating projection models of savings that can be secured by averting expensive, out-of-state placement, Pasti said that New York has done a similar reinvestment. One difficulty they encountered was that some counties had already made substantial progress at the time the reinvestment program was implemented. In those counties, financial benefits were more modest than in places where less had been accomplished.
Murphey said that an indicators report can be a tool in getting the local media to do a story on a particular indicator. One idea he mentioned was getting a local newspaper to run a story on a selected indicator each month.
Keep Holding Up A Mirror
Murphey cautioned against turning indicators work into a high stakes game or a game of "gotcha" with communities -- tying funding to a community's indicators. Vermont tries to recognize improvement whenever they find it.
Stay in This For the Long Haul
This is really long-term work, Murphey warned. It is a different way of thinking and working. It is much longer than any single administration and it is going to take time both at the community level and at the state level. People at the state level and at the community level are often impatient. They want to see results. They are eager to get the data every year, and, if those data don't show that they have made great progress, it can be demoralizing. It's important to remember that there is not a one-to-one relationship between effort and outcome. But success is seen in efforts in which stay the course.
Each of the panel members had a closing comment. Johnson said that he wanted to thank his state partners -- such as Murphey. He said that there is truly a state and local partnership and that there is a level of respect between the state and the local activities. Pasti echoed the idea that it is critical to stay involved for the long haul and that it is an evolving process. Hammond said that, although there is a lot of information available to her in the community profiles, she also seeks information on her own and she keeps a file on where information can be found.