Minutes of the Technical Assistance Workshop, May 3-5, 2000. Session 4: Institutionalizing and Sustaining the Use of Indicators (May 5, 2000)


This session took place on the morning of May 5. Presenters were David Ayer of the Maryland Governor's Office for Children, Youth, & Families (OCYF) and Jim Witherspoon of the Maryland Department of Human Resources. Before the session began, one member of the audience asked that the speakers discuss carrying indicators programs through changes of administration. A second audience member asked that Maryland describe the structure and staffing of their indicators project.

David Ayer

Ayer said that the Maryland presentation on institutionalizing and sustaining the use of indicators would focus on two topics, his talk on communication between the state and communities and Witherspoon's on intergovernmental efforts to institutionalize results and indicators.

Maryland's systems reform effort, and subsequent indicators program, began in the mid-1980s within the human services system and in concert with an Annie E. Casey Foundation demonstration project. Maryland sought to assess how it might better serve at-risk children and families on a variety of health and social needs--then apportioned among a number of agencies, including the Departments of Human Resources, Health and Mental Hygiene, Education, Juvenile Justice, Aging, and Housing and Community Development, and the Office of Individuals with Disabilities--and to create a coordinated system, locally based, that would provide supportive services, not just address crises.

The Subcabinet for Children, Youth, and Families & the Local Planning Entities

Maryland Established a Subcabinet for Children, Youth, and Families to facilitate comprehensive effective and efficient integration of the service delivery system. It also established local planning entities (eventually called Local Management Boards or LBMs) to plan human service delivery in each jurisdiction and provide a central place for local decision making. The LMBs included community members, nonprofits, local elected officials, and human service providers, not just local public agency representatives.

The Task Force on Children, Youth, & Families

During the mid- to late 1990s there was unhappiness in some sectors with system reform and pressure to kill it. Governor Paris Glendening, whose administration had taken over from Governor Schaefer's created a Task Force on Children, Youth, and Families Systems Reform chaired by Lt. Governor Kathleen Kennedy Townsend. One result of the Task Force's work was an invigoration of Systems Reform effort with

  • A focus on results and indicators
  • An expansion of local authority to determine service needs, reaffirming the LMBs
  • The creation of five-year community partnership agreements between jurisdictions and the state

A second result was a reaffirmation of the mission of the system reform effort. In 1989, the Subcabinet defined the mission of child and family services this way:

The mission of services to children and families in Maryland is to promote a stable, safe, healthy environment of ALL children and families, thereby increasing self-sufficiency and family preservation. This requires a comprehensive, coordinated interagency approach providing a continuum of care that is family and child-oriented and emphasizes prevention, early intervention, and community-based services. Priority shall be given to children and families most at risk.

This mission was reaffirmed in the 1996 final report of the Task Force.

Nearly $100 million in state funding was directed by the Subcabinet to the LMBs and community partnership activities (such as Healthy Families and after-school programs).

How the Systems Reform Effort has Driven Maryland to Results and Indicators

The Task Force identified eight (originally nine) results areas. These were:

  • Babies born healthy
  • Healthy children
  • Children enter school ready to learn
  • Children successful in school
  • Children complete school
  • Children safe in their families and communities
  • Stable and economically self-sufficient families
  • Communities supporting family life

A Results Workgroup was established to operationalize these results. The Workgroup included LMB directors and staff, county social services staff members, juvenile justice staff members, public health officials, district school superintendents, state agency officials, and advocacy group representatives. Expert advice in specific results areas was provided by individual members of the Workgroup and by other experts. The Workgroup's meetings were facilitated by staff of the Center for Assessment and Policy Development (CAPD), a not-for-profit organization located in the Philadelphia area. The full Workgroup met ten times over the course of ten months, between April 1997 and January 1998.

Mark Friedman of the Fiscal Policy Studies Institute developed, in conjunction with the Finance Project and the Center for Study of Social Policy, some very practical guidelines for those developing results-based accountability systems. Maryland found Friedman's guidelines on selecting indicators to help operationalize the results extremely useful. The guidelines included these principles

  • The indicator had a known relationship to the dimension of child, family, or community well-being identified in the results area
  • The indicator was reasonably well-measured (that is, it applied to all or most of the relevant population and was collected in ways that supported data reliability and validity)
  • Data on the indicator were readily available at the present time from public sources.
  • Data on the indicator were available at the local jurisdiction level.

Public Engagement

With the help of CAPD, and after a number of sessions, just over 25 indicators were selected. The Work Group's final report, which included recommended results and indicators, was issued. The report was widely distributed and posted on a web site for comment in May 1998. To obtain comments, the report was widely distributed, placed in libraries, and posted on a web site. In addition, a series of local roundtables, hosted by Local Management Boards, were held to enable public review and feedback on the recommended results and indicators. A brochure was developed and distributed to solicit community input and advertise roundtables. A total of 3,850 brochures were mailed to groups and organizations around the state. Public Engagement: Community Input Packets that were prepared for forums held:

  • A copy of the report
  • A sample regional meeting agenda
  • Key presentation points to ensure consistency in reporting
  • A framework for small group discussion appropriate to each of the results areas
  • A standard format for reporting feedback

Twelve roundtable discussions were held with participation from a total of 300 individuals. A broad diversity of individuals participated, including elected officials, librarians, representatives from community organizations, school personnel, health care representatives, LMB members, and others.

LMB directors and participants appreciated having an independent facilitator to guide the discussion and help solicit local input. The material distributed and format of the roundtables was viewed as helpful, particularly the explanation provided for each result and the time dedicated to discussion. Following the roundtables, a statewide public hearing was held in Annapolis, to offer a forum for citizens, organizations, and agencies who could not attend a local roundtable to testify before a panel of partnership members. Participants were asked to present their concerns, as well as submit written testimony.

Summary of Roundtable/Public Hearing

Six result areas remained virtually intact

  • Babies born healthy
  • Healthy children
  • Children successful in school
  • Children completing school
  • Children safe in their families and communities
  • Stable and economically self-sufficient families
  • Added Single Parent Households

Three result areas underwent changes

  • Children enter school ready to learn-preliminary indicators
  • Healthy adults-dropped
  • Communities which support family life-Left for LMBs to develop

The indicators that go with these results include the following.

Babies born healthy. Infant mortality, low birthweight, births to adolescents.

Healthy children. Immunizations, injuries, deaths, substance abuse.

Children enter school ready to learn. Children entering kindergarten with preschool experience, children enrolled in early intervention programs (e.g. Head Start, Infants & Toddlers Program), Low-income children in Head Start or prekindergarten programs.

Children successful in school. Absence from school, academic performance, demonstrated basic skills.

Children completing school. Dropout rate, high school program completion, high school diploma, graduation/school completion of children with serious emotional disturbances

Children safe in their homes and communities. Abuse or neglect, deaths due to injury, juvenile violent offense arrests, juvenile serious non-violent offense arrests, domestic violence.

Stable and economically self-sufficient families. Child poverty, out-of-home placements, single-parent households, Permanent placements, homeless adults and children.

Communities supporting family life. Each LMB to develop its own capacity measures.

Current Organization of Systems Reform

Today, Maryland's systems reform initiative is implemented on the local level through LMBs. Planning is based on local needs researched through a needs assessment process. LMBs are charged with developing a local strategic plan for services. Directors and chairs attended the state's Managing for Results training focused on the state's strategic planning model. The LMBs utilize Maryland's report Results for Child Well-Being to set results for their programs. All 24 jurisdictions have signed contracts with the Subcabinet for the past 3 fiscal years. These contracts fund family preservation and return diversion services, as well as other prevention programs such as disruptive youth, youth suicide prevention, teen pregnancy, prevention, and child abuse and neglect prevention. These contracts are results-based and establish performance measures for each program funded by the LMB. Many and eventually all LMBs will have Community Partnership agreements, multi-year agreements in which jurisdictions extend their reach into result areas in which they need to focus their resources.

LMBs are not service providers. They contract for services with both public and private providers. They receive their funding for these contracts through their individual grant contracts with the Subcabinet. LMB staff members are trained in managing contracts for results. They monitor the service contracts that they fund, and the contracts are developed with set performance measures to hold the vendor accountable. The state's interagency monitoring team conducts evaluations of the contracts, and in conjunction with the LMB establishes a technical assistance plan to ensure that any weaknesses are addressed.

In order to help local jurisdictions, public/private agencies seeking contracts/grants, and the general public with the task of organizing results and indicators, OCYF has taken on the effort to maintain a web page that provides a detailed overview of the selected results and indicators, state level data/graphs, data downloads containing jurisdictional breakdowns, and links to other state and national web sites. The next step for enhancing this site, based on user feedback since it opened in January, is to provide more readily accessible data and graphs at the jurisdictional level.

Simultaneously, the state has been engaged in its efforts to establish a results-based system of accountability at the state level-across all state agencies, not just human services, using the Managing for Results framework, which will be discussed in the next segment of this session.


Christine Johnson sketched the similarities between the situation in Maryland and that in Florida, noting in part that many Florida agencies are moving toward a local focus, but does not have a children's subcabinet.

James Witherspoon

James Witherspoon took up the second part of the Maryland presentation, intergovernmental efforts to institutionalize results and Indicators. His presentation was anchored by a series of slides. These follow.


From Indicators to Strategy to Action

Maryland Managing for Results(MFR)

Managing for Results(MFR)

Managing for Results

Managing for Results

Strategic Planning

The Strategic Chain

Vision & Mission(Purpose)(Draft)

Vision & Mission(Purpose)(Draft)

Guiding Principles(Draft)

Key Performance Areas and Results

Enableing Goals

What are Performance Measures?

Why Performance Measures?

What sre Strategic Focus Areas?

The Role of Strategic Focus Areas

Prioritize and Focus Attention

Identifying Strategic Focus Areas

What are strategic Initiative Teams

What Do Strategic Initiative Teams Do?

Managing for Results(MFR)

Management and Performance

The CQI Approach

Continuous Quality Improvement

Organizations as a Production Process

Measurement and the Production Process

Operational(Managerial) Measures

Activities & Outcomes Sequence Chart

Activities & Outcomes Sequence Chart


MFR and CQI: Holding Agencies and Programs Accountable for Results