U.S. Department of Health and Human Services

Meeting the Challenge of Serving People With Disabilities: A Resource Guide for Assessing the Performance of Managed Care Organizations

Shoshanna Sofaer, Dr.P.H., Sabra F. Woolley, Ph.D., Kyle Anne Kenney, B.A., Barbara Kreling, B.A., and D. Richard Mauery, M.P.H.

Center for Health Outcomes Improvement Research, Center for Health Policy Research, George Washington University

July 1998

PDF Version (93 PDF pages)

TABLE OF CONTENTS

ACKNOWLEDGMENTS

ADVISORY GROUP

I. INTRODUCTION

1. The Context: Meeting the Challenge of Serving People with Disabilities in Managed Care
2. Purpose of the Guide
3. How the Guide is Organized

II. DEVELOPING SYSTEMS TO MEASURE THE PERFORMANCE OF MANAGED CARE IN SERVING PEOPLE WITH DISABILITIES

1. The Population of People With Disabilities
2. What Are the Health Care Needs of People With Disabilities?
3. Performance Measurement: What is It? How Can it be Used?
4. Selecting Performance Measures
5. What it Takes to Build a Performance Measurement System

III. DOMAINS OF MANAGED CARE PERFORMANCE IN SERVING PEOPLE WITH DISABILITIES DETAILED DISCUSSION OF DOMAINS

1. Creating a System With the Right Capacities
2. Providing Access to Needed Services
3. Supporting Member Involvement in Decision Making and System Improvement
4. Resolving Member Problems and Concerns
5. High Quality Interpersonal Interactions Between Members and Providers
6. Using Preventive Services to Keep Members Healthy and Functioning
7. Coordinating and Integrating Medical and Non-Medical Services
8. Using State of the Art Treatments
9. Improving the Outcomes of Care

IV. SPECIFIC MEASURES TO ACCESS THE PERFORMANCE OF MANAGED CARE IN SERVING PEOPLE WITH DISABILITIES

1. How We Identified and Selected the Measures in the Guide
2. Summary Matrix of Domains Addressed in Each Measure
3. Summary of Specific Items Available in the Measures for Each Domain
4. Areas Where Measures Are and Are Not Available
5. Overview of Information to be Provided About Each Measure or Measurement System
6. Descriptions of Measures and Measuring Systems

V. CRITERIA SETS FOR ASSESSING MANAGED CARE ORGANIZATIONS

1. Quality Improvement System for Managed Care (QISMC)
2. Key Indicators from the 1997 Santa Fe Summit on Behavioral Health
3. National Health Law Program (NHeLP) Advocacy Checklist for People With Disabilities

NOTES

REFERENCES AND RESOURCES

APPENDIX: COLLECTING DATA FOR A PARTICULAR PERFORMANCE MEASURE: AN EXAMPLE

ACKNOWLEDGMENTS

The preparation of this Resource Guide would have been impossible without the cooperation of dozens of people who demonstrated their commitment to ensuring the quality of care provided to people with disabilities. In particular, project staff would like to acknowledge the contributions of a group of advisors, whose names are listed below. The group's make-up mirrored our intended audiences, and therefore included staff in State Medicaid agencies and others who work with State Medicaid agencies; staff from managed care organizations; health care providers; and families and other advocates for people with disabilities. A subset of the group provided critical input at the inception of the project by helping us think through the framework for the document; the criteria that should be used in selecting measures; and how the document might best be organized to be useful to different audiences. A second, partially overlapping subset of the advisors actually reviewed the draft document in detail and provided extensive feedback. Of course, none is responsible for the final product, but their comments were both thoughtful and helpful and led to significant changes, and we hope, improvements. One organization, the Center for Health Care Strategies, went above and beyond the call of duty not only by providing input from their staff, but also by supporting Ruth Martin, a consultant to the Center. Ruth Martin made written contributions to the Resource Guide on the important topic of the resources needed to implement an effective performance measurement system.

We were also supported by many experts who had used, reviewed or developed measures and measurement systems that could be considered relevant to the health care needs of people with disabilities. These subject matter experts provided copies of their own materials, referred us to others working in the field, shared their own visions and caveats with us, and shaped our thinking significantly. We are especially indebted to the organizations whose measures, measurement systems and criteria sets are included in the Resource Guide. Not only did they provide us with their materials, but they also helped us present them accurately and completely.

Finally, we would like to acknowledge the continuous encouragement and assistance of staff of the Office of Disability, Aging and Long-Term Care Policy (DALTCP) in the Office of the Assistant Secretary for Planning and Evaluation (ASPE). Mary Harahan, Deputy to the Deputy Assistant Secretary/DALTCP, had the vision and the courage to pursue this project, the type of which is a departure from "traditional" ASPE undertakings. Gavin Kennedy, our Task Order Monitor, was consistently supportive, resourceful and patient. In the final stages of our work, we were pleased to get additional input and advice from Lisa Lang, from ASPE's Office of Health Policy; her expertise in health care quality was especially valuable.

ADVISORY GROUP

Juman Abujbara

Arizona Health Care Cost Containment System

Larry Bartlett

HSR, Inc.

Christy Beaudin

Value Behavioral Health

Allan Bergman

United Cerebral Palsy Associations, Inc.

Karen Brodsky

Center for Health Care Strategies

Carolyn Cocotas

Community Health Plan of the Rockies

Janet Freeze

Maryland Department of Health and Mental Hygiene, Division of Quality Assurance

Annette Hanson

Massachusetts Division of Medical Assistance

Tricia Leddy

Rhode Island Department of Human Services, Office of Managed Care

Trish MacTaggart

Health Care Financing Administration

Barrett Markland

HealthCare Matters, The Managed Care Training and Education Program

Karen O'Connor

Florida Medicaid Quality Assurance

Bobby Silverstein

George Washington University, Center for the Study and Advancement of Disability Policy

Jana Starr

United Cerebral Palsy Associations

Donald Umlah

Arizona Physicians' IPA, Inc.

Mark Wenneker

Boston Community Medical Alliance

I. INTRODUCTION

A. The Context: Meeting the Challenge of Serving People with Disabilities in Managed Care

More and more people covered by Medicaid are enrolling in managed care organizations (MCOs). Under voluntary Medicaid managed care, beneficiaries can choose between an MCO and more traditional Medicaid arrangements, such as fee-for-service (FFS) or the more structured primary care case management (PCCM) program. Under mandatory Medicaid managed care, beneficiaries must enroll in an MCO under contract with their State's Medicaid agency. Most State Medicaid managed care programs began by enrolling people eligible for Aid to Families with Dependent Children, or as it now called, Temporary Assistance for Needy Families (TANF). However, several States have now begun to implement, or seriously consider, the use of managed care for people covered by Medicaid because they are eligible for Supplemental Security Income (SSI); all eligibles in this category are by definition persons with a disability.

Several advantages have been attributed to Medicaid managed care. Proponents believe it will:

• enhance access to high quality health care providers who have previously been unwilling to serve Medicaid eligibles;
• do better at integrating and coordinating care over time and across service delivery settings than the more decentralized FFS system;
• avoid unnecessary and costly care;
• reduce the inappropriate use of service sites, such as hospital emergency rooms, that are often used by low-income people with limited access to primary care physicians; and
• focus more on the provision of preventive health services.

However, others have concerns about whether MCOs can and will meet the health care needs of Medicaid eligibles. They worry, for example, that:

• MCO networks will not include traditional providers of care to Medicaid eligibles (e.g., public hospitals and clinics; community-based health centers and clinics; and inner-city teaching hospitals) who know the special health care needs and personal characteristics of Medicaid eligibles, and are more able and willing to meet those needs; and that
• the financial incentives of capitated payments will cause MCOs to place barriers to access to needed health services.

These concerns become more significant when decisions are being made about the use of MCOs to care for people with disabilities, many of whom need a complex (and sometimes uncommon) mix of health and social services to maintain health and functioning. One advantage of MCOs is that it is easier to measure, and thus to improve, the performance of a given MCO than it has been to measure, and especially to improve, the performance of a community's FFS system. The expansion of managed care has been accompanied by increased attention to performance measurement for health care delivery systems, and an increased emphasis on the need to hold systems accountable for their performance.1

While there is little evidence that Medicaid FFS as a system took very good care of the population of persons with disabilities, it is clear that specific health professionals, facilities and programs have provided excellent and responsive care to particular patients in particular communities. For many, the ultimate value of managed care will be determined by whether it works well for people with complex and specialized health care problems and needs, the kind of problems and need of many persons with disabilities. As more and more people with disabilities enroll in managed care, it is critical, for us all, to determine whether MCOs are performing well in meeting their needs.

Comprehensive systems for measuring the performance of health care systems in caring for persons with disabilities do not yet exist. At the same time, thousands of individual quality measures exist but it is often difficult to discern which will be most reliable and relevant to measure MCO performance in caring for people with disabilities. In these circumstances, it is easy to respond either by (1) doing little or nothing to measure MCO performance or (2) mounting costly efforts to measure hundreds of highly specific aspects of quality that may fail to provide a coherent picture of performance. This Resource Guide is designed to help those who want to begin to work toward a comprehensive system, today, by using measures available right now that have a clear relationship to domains of performance important to the care of persons with disabilities.

B. Purpose of the Guide

This Resource Guide is designed to support efforts to measure and improve Medicaid MCO performance for people with disabilities. It presents the results of an extensive search for existing performance measures that can be used for quality measurement and improvement by:

• State Medicaid agencies;
• managed care organizations themselves;
• providers within MCOs; and
• people who advocate for the health care needs of those with disabilities.

The purpose of the Resource Guide is threefold:

1. To provide an overview of the issues to be considered in measuring the performance of MCO in taking care of people with disabilities;

2. To bring together in one volume key information about measures that now exist, or are about to be made available, that can be counted on to support efforts to measure and improve the care provided by MCOs to people with disabilities; and

3. To make clear the areas where technically strong measures do not exist, in order to encourage further measurement development in these areas.

The Resource Guide is the result of a ten-month effort by the Center for Health Outcomes Improvement Research2 at the George Washington University Medical Center. This effort was supported by the Office of Disability, Aging and Long-Term Care Policy of the Office of the Assistant Secretary for Policy and Evaluation (ASPE) at the U.S. Department of Health and Human Services. ASPE commissioned the Resource Guide in response to numerous requests from the field to disseminate, sooner rather than later, the best available measures of performance in caring for people with disabilities.

Two important features of the project must be noted. First, the project was not designed to develop new measures. Rather, it identified existing measures, or measures that are about to be released. The time and resources available were not sufficient for the development of new measures. It is relatively easy to identify a dimension of performance for MCOs, and even to reach consensus across stakeholders that this dimension is significant to meeting the needs of people with disabilities. It is far more difficult, and takes considerably longer, to develop a reliable and valid way of measuring a dimension of performance. The project has helped to identify the gaps in existing performance measures, i.e. the dimensions of performance that many people think are critical but for which valid and usable measures are not available. It can, therefore, serve as a guide to future measurement development efforts.

A second feature of the project is that it did not focus on measures that would be used to assess the care provided to people, such as the frail elderly, who are in nursing homes or other residential long-term care facilities. Again, given time and resource limitations, the project's goal was to concentrate on care delivered to other populations of Medicaid eligibles with disabilities who are being enrolled in MCOs.

C. How the Guide is Organized

Following this Introduction, the Resource Guide has four chapters. Chapter Two: Developing Systems to Measure the Performance of Managed Care in Serving People With Disabilities, discusses issues that should be considered in the selection and use of performance measures, including:

• the characteristics of people with disabilities;
• the health care needs of people with disabilities;
• what performance measurement is, who can use performance measurement and how; and
• what it takes to build an effective performance measurement system.

Chapter Three: Domains of Managed Care Performance in Serving People with Disabilities, presents a framework for categorizing the aspects of MCO performance considered important in providing quality care to people with disabilities. This Chapter should be read by everyone. The framework it presents performs two roles:

1. It presents the full range of topics that are important to address with respect to serving people with disabilities.

2. It serves as an organizing tool for describing and categorizing particular measures; the framework will be used for that purpose throughout the Resource Guide.

In Chapter Four: Specific Measures to Assess the Performance of Managed Care in Serving People with Disabilities, presents measures and measurement systems that deserve serious consideration for inclusion in a performance measurement system. We are not recommending that anyone use ALL these measures. Rather, they are a good range from which to select a sub-set for implementation. This Chapter includes:

• a discussion of how we selected the measures;
• a specification of the information that will be provided about each measure or measurement system;
• a matrix that indicates what domains are addressed in each measure or measurement system in the Resource Guide;
• a summary, for each domain, of the specific measures available; and
• a detailed description of each measure or measurement system.

The Resource Guide does not include copies of actual measures, such as complete surveys, or the full technical specification of measures. The inclusion of such material would make the document large and cumbersome. Rather, the Guide provides sufficient information on each measure to help potential users decide whether further consideration of the measure is warranted, as well as information about where and how to get actual instruments, technical specifications, other documentation and in several cases technical support.

Chapter Five: Criteria Sets for Assessing Managed Care Organizations, presents material that may also be of use in examining the performance of MCOs in serving people with disabilities. Several groups have identified specific characteristics of MCOs that they think have important consequences for the care of people with disabilities. These are criteria sets, rather than formal technical measures. That is, no method has been specified for collecting information to determine whether a characteristic is present or absent. In some cases, determining whether a characteristic is present may be relatively easy; in other cases, it may be quite difficult. We have included three criteria sets in the Resource Guide because:

• unlike most technical measures, they have been developed specifically to address the concerns of people with disabilities;
• they can often be used to examine MCOs prior to finalizing contracts; and
• they provide a starting point for the development of more formal measures and data collection efforts.

Finally, a list of References and Resources provides information on other resources and documents that you might find useful in developing a performance measurement system.

II. DEVELOPING SYSTEMS TO MEASURE THE PERFORMANCE OF MANAGED CARE IN SERVING PEOPLE WITH DISABILITIES

A. The Population of People With Disabilities

To determine how best to assess the performance of MCOs in serving Medicaid eligibles with disabilities, it is important to understand key characteristics of this population. When the last U.S. Census was conducted in 1991-1992, there were 48.9 million Americans with a disability, or nearly 19.4 percent of the total U.S. population (McNeil, 1993). At that time, approximately 19 percent of persons with disabilities aged 15 to 64 were covered by Medicaid.

In 1995, according to the Health Care Financing Administration (HCFA), approximately six million people with disabilities were covered by Medicaid, of whom 1.3 million were children. Within Medicaid, those with SSI were the second fastest growing eligibility group between 1990 and 1994, increasing at an average annual rate of 10 percent (Davis and O'Brien). These authors also report that as of 1995, the disabled "make up only 15 percent of all Medicaid users, while their spending accounts for 39 percent of all program payments."

The 1994 National Health Interview Survey (NHIS) included a supplement designed to learn more about people with disabilities. Utilization of medical care among the Medicaid SSI population was slightly higher than utilization in the general Medicaid population, but similar to utilization for all people with disabilities. The survey reveals that many children with disabilities are eligible for Medicaid not under the SSI, but under TANF. This implies that measuring the performance of MCOs in serving people with disabilities, and especially children with disabilities, may be a significant concern even when a Medicaid program does not enroll SSI-eligible persons in MCOs.

Analysis of the NHIS also reveals that estimates of the size of the population with disabilities vary widely, depending upon the definition used. Some respondents who would be classified as having a disability under certain definitions (e.g., impairments, disease conditions, inability to perform certain activities of daily living) do not self-identify as having a disability. This may reflect differences in attitude between people with disabilities and their health care providers. As Harahan noted in a discussion of these results, people with disabilities "may demand control over their own lives and maximum choice."

B. What Are the Health Care Needs of People With Disabilities?

As already noted, the extent and nature of health services needed by people with disabilities varies widely. This section presents the health care needs of three subgroups: healthy people with disabilities; people with disabilities who have ongoing but not particularly complex health conditions; and people with disabilities who have complex and uncommon health conditions. Through this discussion, we highlight the issues that must be addressed in measuring the performance of MCOs in caring for a population of people with disabilities.

Taking Care of Healthy People with Disabilities

People with disabilities need the same set of preventive and curative health services as those without disabilities. Indeed, as in the general population, many people with disabilities need regular preventive care but only episodic curative care, since they are basically in good health.

In providing health services even to healthy people with disabilities, however, delivery systems must identify, and take into account, the particular challenges facing each person. For example, HCFA data cited by Davis and O'Brien (1996) reveal that "Mental impairments, including mental illness, mental retardation, and developmental disabilities, predominate among both the adult and child SSI populations." People with these conditions comprise 67 percent of adult and 57 percent of child SSI recipients. When preventive and curative medical care is being provided to this group, it must be tailored to take into account their psychological, social and developmental needs and problems. This implies two special needs even for the physically healthy person with a mental or developmental disability:

• the need for coordination of care provided by physical and behavioral health care providers;
• the need for health professionals who do not specialize in behavioral health to understand how mental health and developmental problems can affect the prevention, diagnosis, treatment and follow-up of other medical conditions.

A wide variety of other diagnoses and conditions can result in a disability that makes one eligible for SSI. Medicaid SSI eligibles also include people who have sensory impairments; neurological problems; mild to very severe limitations in their motor functioning; and respiratory limitations. As in the case of those with mental impairments, the physical, psychological and other challenges presented by their particular condition must be taken into consideration in ensuring access to medical care and in ensuring care will be appropriate to the individual. Access and appropriateness are concerns for everyone. Issues that are often of special concern to persons with disabilities include:

• facilities and equipment that do not present physical and architectural access barriers;
• specialized transportation to care sites;
• geographic proximity of health care facilities, with in-home care needed in some cases;
• alternative methods of communicating and interacting with enrollees that are appropriate to those with sensory or linguistic impairments; and
• active participation of the individual people with disabilities and where appropriate his/her family or other caregivers in decisions about their care.

Ongoing Health Care for Persons with Disabilities

Other persons with disabilities resemble (indeed may be) persons with chronic medical conditions. That is, in addition to regular preventive services and episodic curative care, they need specific services on a regular basis for an ongoing condition. They (and often their family) must often participate actively to make sure their condition is well managed. Since their condition may never be "cured," the focus of attention shifts to their ability to live as full and active a life as possible is maintained, and to avoid preventable deterioration and complications of their condition. For this group of persons with disabilities, the following needs become especially important:

• access to a regular health care provider (whether a primary care provider or a specialist) who understands their particular problem and how it is likely to affect their life;
• ability to maintain a relationship with this regular health care provider over time;
• careful tracking by the health care system of the individual's condition, of their response to treatment, of changes in their life circumstances and daily functioning; and of the emergence of other medical problems which can affect, or be affected by, their ongoing condition;
• access to patient education, nutrition, exercise and other services that support effective management of the underlying condition;
• access to services needed to prevent complications or deterioration or to treat them promptly when they do occur;
• access to health services that can maintain or enhance the person's independence and improved functioning in daily life;
• links to non-medical services (e.g., housing, education, employment training and placement, transportation, etc.) that also support independence and improved functioning in daily life;
• active involvement of the person and where appropriate his/her family in decisions about treatment alternatives and in the implementation of treatment plans; and
• support for family members who care for or are simply significantly affected by the health and functioning of a person with an ongoing and disabling condition.

Caring for People with Complex and Uncommon Needs

Compared to the general population, a higher proportion of people with disabilities face quite serious and often uncommon medical problems, that must be treated or managed effectively, and that can significantly influence the treatment of other medical problems that occur over their lifespan. Their health care needs are complex and specialized. Multiple providers may care for an individual patient at any given point in time. The specific services required may change over time. Given these circumstances, the following issues, in addition to those already mentioned, become particularly important:

• access to highly specialized providers and facilities experienced in diagnosing and treating the individual's condition;
• access to rarely performed tests and procedures;
• access to rarely prescribed medications, medical supplies and equipment;
• coordination of care across providers;
• continuity of care over time; and
• careful tracking of changing circumstances and needs as well as relevant medical advances.

Most MCOs are not organized, at the outset, to take care of the full range of complex and uncommon medical problems that may be faced by their members. Their network may not include the required specialists. Their formulary may not include the required medications. MCOs that agree to take responsibility for a population of persons with disabilities need to expect the unexpected, and be ready, willing and able to respond. The responsiveness of MCOs to complex and uncommon medical problems is influenced by the following factors:

• the use of a reasonable definition of medical necessity that does not exclude services essential to diagnosing and treating uncommon conditions;
• the willingness and ability to add highly specialized providers and facilities to their network in order to diagnose and treat uncommon conditions;
• the willingness and ability to provide medications not included on their formulary when essential to treating uncommon conditions or preventing complications;
• the availability of user-friendly, well-understood methods that members can use to appeal denials of care and get prompt responses; and
• the availability of care coordination services.

C. Performance Measurement: What is It? How Can it be Used?

This section of the Resource Guide addresses how to think about and plan for a system for measuring MCO performance in serving people with disabilities. It begins by defining performance measurement, and then describes how performance measurement can be used by different groups, and what resources are needed to use it well. The section ends with a discussion of how to enhance, over time, our efforts to measure and improve performance.

What is Performance Measurement?

In this Guide, performance measurement is defined as:

the process of using formal, scientifically grounded tools and methods to collect information about a health care delivery system, such as an MCO, to determine whether its characteristics and actions, and the consequences of its actions, meet expectations.

Performance measurement is a key element in holding health care delivery systems accountable for what they do. It is intended to inform decisions and guide actions. It is not measurement simply for the sake of measurement, or to conduct research.

As implied in our definition, performance measurement involves comparing performance against a set of expectations about what an MCO can and should be doing for its members. These expectations come in part from what research tells us can and should be done to achieve desired outcomes. However, the performance measures we choose, and in particular what we choose to measure, invariably reflect, in addition, values and preferences.

This leads to a question: whose expectations, and whose values and preferences, should drive the development of performance measures? The development of effective performance measurement systems will require participation from many different stakeholders: purchasers (such as State Medicaid agencies); policy makers; MCOs; health and social service providers; and consumers, including persons with disabilities and their families and other informal caregivers. Each of these stakeholders is likely to have distinct values, preferences and expectations. To develop a performance measurement program, it is essential to develop sufficient consensus about expectations. While it is difficult to reach complete agreement on expectations, it is clearly possible to identify a set of important and widely shared expectations.

How Can Performance Measurement be Used?

Performance measurement can be undertaken by different organizations, for different purposes. We will discuss how measuring performance of MCOs in serving people with disabilities might be used by four different and significant constituencies: State Medicaid agencies; MCOs; people with disabilities and those who care or advocate for them; and health care providers.

State Medicaid agencies can use performance measurement to:

• identify areas where improvement is needed in the performance of particular MCOs, set goals for improvement and in some cases incorporate them into contract specifications;
• identify exemplary behavior in particular MCOs to highlight and disseminate, or use as a benchmark for the performance of other plans;
• provide useful and reliable comparative information to Medicaid eligibles who are trying to choose among various MCOs they can join;
• select an MCO for persons who do not explicitly choose one for themselves;
• decide whether to contract with a given MCO in the future;
• determine if there should be a ceiling set for new enrollments in a given MCO;
• in extreme cases, introduce intermediate sanctions or even terminate a contract with a plan; and
• identify patterns of poor performance across multiple plans that may signal the need for broader changes in policies, procedures or even legislation.3

Managed care organizations can use performance measurement to:

• identify where they need to focus internal quality improvement activities;
• track their progress over time in making quality improvements;
• make decisions about which provider groups or facilities it wants to include in its network in the future;4 and
• market themselves to potential enrollees.

People with disabilities, their caregivers and advocates can use performance measurement to:

• make more informed choices both between managed care and FFS systems and among various MCOs;
• become more capable of making their own choice, instead of being arbitrarily assigned to an MCO by the State Medicaid agency;
• identify and publicize significant problems either in a particular MCO or across several MCOs, in a way that moves beyond individual anecdotes;
• identify and publicize positive findings to highlight what can be achieved for people with disabilities, how it can be done, and who is doing it.

Health care providers (such as hospitals, health centers, individual physicians and physician groups) can use performance measurement to:

• support their own quality improvement efforts;
• highlight for the public the things they do best; and
• decide which MCO networks they would like to join or remain a part of.

Performance Measurement and Comparisons

Performance measurement always involves making comparisons. Here are some comparisons that can be made using performance measurement:

• comparing different MCOs to each other:

  who is performing better at ensuring that patients are actively involved in decisions about their care? who is not doing as well?

• comparing each MCO's performance to an explicit standard:

  do people get seen for urgent care within 12 hours? do at least 90 percent of children get all required immunizations by the age of two? are sign language interpreters always available for regularly scheduled appointments with hearing impaired persons?

• comparing how an MCO performs at a certain point in time to how they performed at another point in time:

  is the MCO reducing how long patients wait in the office before they are seen? or are patients actually waiting longer?

• comparing the performance of MCOs in caring for people with disabilities to their performance in caring for other members:

  how do rates of mammography compare between women over 50 with and without disabilities? how often do people with disabilities disenroll from a given MCO because of quality concerns, as compared to disenrollment among those without disabilities?

• comparing the performance of MCOs, across the board, with the performance of other delivery systems such as FFS or PCCM:

  how do people with disabilities who join MCOs rate their access to specialists, as compared to those who remain in FFS or PCCM?

Different kinds of comparisons reflect, sometimes implicitly and sometimes explicitly, different expectations. For example, when MCOs are being compared to each other, they are implicitly measuring their performance against the "average" performance in the group. Saying that a particular MCO is "better than average" may be misleading if everyone is performing poorly. Saying performance is "just about average" may similarly be misleading is everyone is performing very well.

On the other hand, comparing each MCO's performance to a standard makes expectations very explicit. Standards, like measures, have to be developed. They can be derived from:

• what can actually be achieved because at least some pathbreaking MCOs have achieved it (this is often called a benchmark):

  for example, a small group of MCOs has found methods to ensure that primary care physicians diagnose depression and make appropriate referrals where needed for 80 percent of their patients

• what research shows is essential, or critical, to achieve desired outcomes:

  for example, when 90 percent of a population has been immunized, the population as a whole achieves "herd immunity" reducing to virtually nothing the chance of disease transmission

• what stakeholders agree is both achievable and morally right:

  for example, a standard can be set that no one should face physical barriers to access to health care facilities

Finally, when the performance of MCOs in serving people with disabilities is compared to their performance in serving other members, there is an implicit expectation that people with disabilities deserve, can, and should get at least the same quality of care as everyone else.

D. Selecting Performance Measures

There are innumerable attributes of an MCO that can affect the health, functioning, quality of life and satisfaction of the people it serves. There are thousands of measures of the quality of medical services, most designed to determine whether, for a patient with a particular diagnosis or condition, the correct services were provided at the right time by the appropriate people and achieved the desired consequences. But very few MCO attributes, and very few condition-specific quality measures, are likely to work well as part of a system of performance measurement. Why? For one thing, no one has the resources to measure everything that could be measured. No one has the time or resources to use all the information that would be generated by such a massive measurement effort. Indeed, it is important to avoid spending so many resources on performance measurement that it detracts from, rather than adds to, our ability to provide good service to people with disabilities.

To build a performance measurement system, it is critical to select a limited number of good measures that together provide a coherent picture of a health care delivery system, rather than measure everything. As noted earlier, the performance measures selected will get the attention of MCOs, so it is important to select the right set. Here are some criteria that can be used to select measures; they were also used to select the measures included in this Resource Guide:

1. The measure should tell you about something that is considered of great significance.

   As noted above, performance measures reflect expectations and values. Some things that an MCO does (or does not) do are far more significant that others, depending on your expectations and values.

2. The measure should address an aspect of the performance that the MCO can significantly influence.

   Performance measurement is a tool to hold MCOs accountable. It is neither sensible nor fair to hold an organization accountable for what it cannot influence. Notice this criterion uses the term influence rather than control. No organization can control anything completely. However, MCOs have significant influence over many things they do not control. Another way to put this is this: does the measure address something that is actionable, something that the MCO can work to improve? does it give the MCO clear direction about what it needs to work on?

   Related to this criterion is the issue of what the MCO is contractually required to provide to its members, and in particular to its members with disabilities. Thus, for example, if a contract between a State Medicaid agency and an MCO does not specify that a given service must be provided to enrollees, it may not be appropriate to measure whether or not that service is indeed provided.

3. The measure should tell you something that reflects the performance of an MCO as a system of care, rather than a very narrow aspect of its functioning.

   This criterion is somewhat difficult to understand and apply. An example may help. When health care delivery systems in different countries are compared, one of the performance measures that is almost always used is the infant mortality rate. This measure meets our first two criteria well: increasing the number of newborns who survive to at least the age of one is clearly important; and while the health care delivery system cannot control all the factors that influence infant mortality, it has a significant influence on many others. It also meets this third criteria, because reducing infant mortality requires that many different parts of the health care system do a good job. The infant mortality rate reflects whether women get prompt access to pre-natal care; whether the care is consistent and effective; whether the most up-to-date tests and treatments are being used; whether potentially high-risk pregnancies are being identified early; and whether action is being taken to reduce or address risks swiftly and effectively.

   Another way to think about this criterion is in terms of whether a particular action that an MCO can take would have a big effect on the entire experience a member or a patient has in the MCO. For example, whether a new member of an MCO is linked to a primary care provider quickly (say within 30 days) could be critical for their ability to use and benefit from the entire system. Another example of special importance to people with disabilities is whether the MCO has an efficient and timely process for bringing new, highly specialized providers into their network when a new member enrolls who needs such providers to maintain or improve health and functioning.

4. The measure should let you make the comparisons you want to make.

   As discussed above, performance measurement always involves comparisons. Measures should be selected that are relevant to the comparisons being made. Just as important, measures have to be selected that can be implemented across all the organizations or systems being compared. In particular, the data needed for the measure has to be available across all these organizations and the measures need to be sufficiently specific to ensure that data are collected exactly the same way.

5. The measure should address an event that can be observed often enough to produce reliable results.

   Most performance measures are rates; they measure how often, or for what proportion of people, an event takes place. The denominator of a rate indicates the maximum number of times, or the maximum number of people, for a given event. For example, the denominator for a disenrollment rate would be the number of people enrolled in a plan over a given time period. The numerator of a rate indicates the actual period of time, or the actual number of people, for a given event. Thus, the numerator for a disenrollment rate would be the number of people in a plan who disenrolled (typically for specific reasons related to their dissatisfaction with care) in that same time period. When the denominator or the numerator are likely to be quite small, it becomes difficult, for statistical reasons, to have confidence in the reliability of a rate. It also becomes very difficult to make comparisons.

   This "small number problem" makes it difficult to include as performance measures events that are specific to a particular health problem or condition, especially if that condition is not very common in the general population, in the population of people with disabilities, or in the population of people enrolled in a particular MCO or group of MCOs. For this reason, we have emphasized, in the Resource Guide, measures that are applicable to larger groups of people, rather than condition-specific measures.

E. What it Takes to Build a Performance Measurement System

Measures are not the only resource needed to build, and to use well, a system of performance measurement. In this part of the Resource Guide we will discuss the other resources that need to be in place to build a system.

Planning and Designing a System of Performance Measurement

As noted above, performance measurement requires that we clarify expectations. Also as noted, many constituencies have an important stake in what gets measured, how it is measured, how it gets reported, and to whom. For this reason, it is highly desirable to design systems of performance measurement with input from multiple stakeholders. In most but not all cases, the lead in development of such systems will come from State Medicaid agencies. The lead organization should convene all stakeholders, including other State agencies who provide health and related social services to people with disabilities. Experiences in several States indicates that the initial planning and design process can take up to a year. The convenor needs to have skills in planning, in facilitating group processes and in managing conflict, as well as technical knowledge. Some may find it useful to contract with an external consultant as a facilitator, since an outsider may appear more neutral than any of the parties.

Resources Needed to Collect and Analyze Performance Data

No matter how carefully measures are selected to assess MCO performance in caring for people with disabilities, the system will fail unless accurate and comparable data are collected in a systematic manner and unless it is analyzed and interpreted correctly. Two key players, State Medicaid agencies and MCOs, are most likely to be directly involved in data collection and analysis. Two kinds of data are likely to be collected for performance measures: primary data, collected for example through surveys of members; and secondary data, collected from existing administrative and clinical records and information systems.

State Medicaid Agency

Within State Medicaid agencies, a critical resource is people. One or more dedicated staff members will be needed with knowledge of the following:

• performance and quality measurement;
• the strengths and weaknesses of different approaches to data collection;
• basics of survey research and sampling;
• contracting and coordinating with third-party vendors to conduct surveys;
• drawing survey samples using State and MCO records;
• the structure, potential and limitations of State and MCO management and clinical information systems;
• methods for auditing and validating data provided by MCOs;
• data tracking and management;
• alternate approaches to the transformation of raw data into rates and other findings;
• methods for risk adjustment of performance and quality data;
• comparative analysis of data; and
• presentation of data to multiple audiences.

States need to decide how they want to receive performance information. There are three basic choices:

1. MCOs submit the calculated performance measure;

2. MCOs submit the data required to calculate the numerator and denominator of the performance measure and the State calculates the measures;

3. for measures calculated from administrative data only, the MCOs submit all claims or encounter data and the State calculates the measures.

The approach chosen has implications for the extent and nature of computer hardware and software resources, and for how many and what types of staff are needed to receive, clean, and load the data and calculate the performance measures.

States must also choose how they will audit and validate data. The need for validation increases as the MCO, rather than the State, takes responsibility for data collection and the calculation of performance measures. But much of the data will inevitably be based on MCO records. Unless the data are audited, the State cannot assess the level of confidence that it should have in the reported performance measures or correct the problems with the performance measures. There are several ways in which the State can audit the data:

• using State staff;
• using a contractor, such as the External Quality Review Organization (EQRO); or
• requiring that the MCOs hire independent auditors to certify their performance measures.5

Managed Care Organizations

To participate effectively in a performance measurement system, MCOs need three broad categories of resources:

• staff with appropriate knowledge and analytic skills;
• an appropriate management/medical information system; and
• staff with appropriate skills to collect data from clinical records.

Specific staff expertise includes:

• performance and quality measurement;
• how to translate the definition of a performance measure into detailed operational specifications for data collection;
• an understanding of MCO program operations, including related quality assurance and quality improvement activities;
• and understanding of the potential and limitations of the MCOs' management and clinical information systems (and the quality of information systems at the provider level);
• the history of data collection and information systems to identify changes that would affect the reliability of trend data;
• data collection methods, including abstraction of data from non-electronic medical records;
• data tracking, cleaning and management; and
• auditing and validating methods.

If MCOs choose to implement performance measures based on surveys, they will also need either an in-house survey capacity or more likely the capacity to select and work with an outside vendor.

As performance measurement and reporting have become more critical for MCOs, the need for management information systems that can support these efforts has grown. In an ideal world, all medical records would be automated and MCOs would have direct access to clinical data that could be combined with administrative data, such as enrollment records, to produce timely, accurate performance measures. Unfortunately, current reality is far from ideal. Therefore, clinical information is derived, to the extent possible, from claims and/or encounter data. Information is further limited by the type of coding used for office visits and procedures and the coding used for diagnosis. For Medicaid members, enrollment data come directly from the State and are not under the control of the MCO; these data are sometimes critical to specifying the population for which a measure will be relevant. Therefore, collaboration at the operational level will almost always be needed between the State Medicaid agency and MCOs in order to generate many performance measures.

An MCO's ability to participate in an effective performance measurement system depends on its having a management information system (and related staff) that:

• Collects and reconciles enrollment data and maintains an enrollment history;
• Collect and processes claims/encounter data in a timely and accurate fashion and maintains a claims history;
• Can integrate data from other sources, such as pharmacy claims data from a third-party vendor; and
• Has a flexible report-generator or can export data to a report generation program in order to calculate the performance measures.

Many performance measures will require the use of clinical information (that may well not be computerized) as well as administrative data (that is more likely to be computerized).6 This means it is important to have staff with expertise and experience in the abstraction of medical records. As important is a healthy relationship between the MCO and its providers. Especially in more loosely organized and decentralized provider networks, the MCO will need to work carefully with its providers to ensure that data are reliably and consistently collected. In this context, auditing and validation are even more important.

Level of Resources

The amount and type of data collection and analysis resources needed by States and MCOs depend on the number and type of performance measures chosen, the frequency of reporting, and the level of statistical confidence needed. In general:

• More frequent reporting requires more resources, unless data are readily available from a computerized management information system.
• Measures that require manual data collection (e.g., medical records review) require more resources than measures that can be calculated directly from computerized records.
• When sampling is used to collect data to calculate measures (for example with manual medical records review or surveys), higher statistical confidence levels require more resources than lower confidence levels.
• If different samples have to be drawn for multiple measures, more resources are required. This can be somewhat mitigated when data for several measures can be collected from a single medical record.
• When oversampling or screening is used to ensure that a sufficient number of people with disabilities are included, higher levels of resources are needed.

Resources Needed for Disseminating Performance Information

All too often, based on traditional practice, performance measurement is viewed as of interest only to technical professionals. In fact, however, a critical value of performance measurement is that it promotes both general accountability to the public and the generation of information that can support decisions and actions made by individual members of the public. Performance data can only be used by those who have access to it. In planning a performance measurement system, attention needs to be given to whether, how, when and to whom performance data will be disseminated.

MCOs and State Medicaid agencies may find themselves uncomfortable at the thought that they will not be the only people who see, interpret and use performance information. It does "raise the stakes" both on the relevance and quality of the data when wider dissemination is planned. In some ways, however, this a good reason to pursue dissemination: the quality of data may rise when people know that the public will see it.

The implication is not only that dissemination needs to be considered in planning, but also that resources will be required to develop and implement the dissemination strategy. The resources include staff or third-party consultants and vendors with skills and experience in:

• communication, including writing and graphic design;
• the use of formal and informal media and channels for reaching target audiences; and
• tailoring the presentation of data to the characteristics and interests of the audience, including their literacy levels, cognitive and sensory impairments, language preferences, and trusted information channels.

People with disabilities, their caregivers and advocates are not just another audience in this context; they understand best how they can be reached effectively and it is wise to involve them from the outset in planning dissemination efforts.

Enhancing Efforts to Measure and Improve Performance

At the outset, State Medicaid agencies and MCOs will not have all the resources needed to implement an ideal performance measurement system, including in particular:

• skilled and knowledgeable staff;
• effective systems for collecting and analyzing data;
• methods for disseminating the results of performance measurement; and
• an accepted forum where various stakeholders can participate in the process of planning and trouble-shooting.

Performance measurement systems will be "works in progress" for many years. They will not be ideal and should not be expected to be ideal at the outset. This is one area where the adage that "the perfect can be the enemy of the good" is very applicable.

The development of performance measurement systems has to start somewhere. We strongly recommend beginning with a limited set of measures that are scientifically well-grounded, relevant to the concerns of stakeholders, and feasible to implement given the current state of available data collection and information systems. The planning process should be designed to help you identify these measures. It will also serve to identify the more basic structural barriers to using performance measurement. Often, simultaneous work may be needed on structural barriers and the first stages of building a performance measurement system.

Experience can be a harsh but excellent teacher. For this reason, we also recommend learning about and staying in touch with others who are embarked on similar efforts, so you can gain from their experience as well as your own. The reverse is true: share your experiences with others.

The long-range development of performance measurement systems clearly requires investment of resources. Acquiring and maintaining those resources also means that the benefits of performance measurement be made clear to those with influence over resource allocation. Ultimately, however, development of an effective performance measurement system will require that senior officials in State Medicaid agencies, MCOs, in the community of people with disabilities, and among health care providers, make a commitment to using objective information to assess and improve our systems of care.

III. DOMAINS OF MANAGED CARE PERFORMANCE IN SERVING PEOPLE WITH DISABILITIES

This Chapter presents a framework of domains of managed care performance that the project identified as critical to serving people with disabilities well. These domains permit measures to be categorized into broad topics that have been identified as important to serving the needs of people with disabilities. These domains and the example measures we present in each were based on:

• interviews with numerous experts and stakeholders;
• review of research on what people find important about their health plans; and
• analysis of many documents (see References and Resources).

The framework represents a synthesis, by project staff, of many sources of information, including existing approaches to categorizing measures. For example, quality measures were first categorized by Donabedian into three types: structure, process and outcome. Many people are familiar with this framework, but it does not address the actual content and substantive issues addressed by measures. It is therefore less useful for those trying to think about performance measurement and to select specific measures.

This framework of domains represents an ideal: it would be highly desirable to find well-grounded and usable performance measures across all the domains. However, our search for measures indicates that some domains are thinly populated compared to others. Nevertheless, it is useful to have a comprehensive set of issues and concerns to consider, even if measures do not exist today. By starting with a comprehensive set of domains, it has been possible to identify clearly the gaps in existing measures, thus providing direction for future measurement development efforts.

The nine major domains in our framework are presented first. Details and discussion of each domain follow.

Detailed Discussion of Domains

A. Creating a System with the Right Capacities

As noted above, quality measurements have been distinguished in terms of structure, process and outcome. The earliest quality measures emphasized structure and were typically used to give a facility a license or to accredit a program. This first domain of MCO performance also emphasizes structure, but we use here the broader term capacity. One great advantage of structural or capacity measures is that they can be used prospectively, that is before people with disabilities are enrolled in a particular MCO. They can also be used even after enrollment, to make sure that capacities are in fact present and perhaps being enhanced. Here are examples of indicators of the capacity of an MCO to take care of people with disabilities. As with all the lists in this section of the Resource Guide, these examples are not considered to be complete; you may well identify additional indicators and measures, as we have.

• The number, mix and qualifications of providers included in the MCO network;
• The stability of the MCO provider network;
• The presence of procedures for adding new providers to meet unexpected needs of members;
• The presence of well-designed and active quality assurance and quality improvement systems;
• The presence of a system for assessing and tracking the health and functional status of members (especially those with disabilities); and
• The presence of a method for coordinating the care provided to members (especially those with disabilities) with complex or exceptional needs.

Note that in these examples, the emphasis is on whether the capacity is in place. We view this as a necessary, but not sufficient, condition for high performance. It is also important to determine whether these capacities are actually used and whether or not they are effective. These issues are addressed in other domains.

B. Providing Access to Needed Services

Access to care has long been a critical element of any assessment of a health care delivery system. When access barriers exist, people have less chance to get the services and care they need. Here are examples of indicators of access to needed services that are of particular relevance to people with disabilities; many are important to anyone.

• Facilities, medical equipment and communication equipment accessible to those with physical challenges, including visual and hearing impairments;
• Availability of transportation to care sites;
• Availability of convenient times of service;
• Timeliness of access to routine and urgent care (e.g., waiting time for appointments; waiting time after arrival);
• Access to needed tests and treatments without extensive delays or hassles waiting for prior approvals;
• Access to needed prescription medications (including medications not included in the MCOs formulary, although they are on the State's Medicaid formulary);
• Access to identified/selected primary care providers;
• Access to specialists;
• Access to services of special importance to persons with disabilities (e.g., durable medical equipment and provision for the prompt repair of such equipment; rehabilitation services; physical, occupational and speech therapy; home health care (both skilled and unskilled); respite care); and
• Availability of translation and interpretation services for non-English speakers (including people with speech and hearing disabilities who use American Sign Language).

C. Supporting Member Involvement in Decision Making and System Improvement

Increasingly, Americans want to be more active in their interactions with the health care system. People with disabilities have been pioneers in pursuing a more active role and greater autonomy. In this domain, there are two levels at which it is important for MCO members with disabilities to be involved in decision making and system improvement. The first level is the individual patient's interaction with the MCO and their health care providers. The second level is the policy decisions that affect or guide a given MCO, or the entire program of care for people with disabilities.

Involvement in Decisions and Improvements at the Individual Level

• Provision of understandable information on how to get needed services and resolve member problems;
• Provision of understandable information on methods for filing complaints and grievances;
• Provision of understandable information on the definition of medical necessity used by the plan, on the criteria applied to determine medical necessity, and on the process used in making medical necessity decisions, including the process for appealing such decisions in a timely manner;
• Involvement of members in treatment planning for management of ongoing conditions;
• Provision of information on all treatment options; involvement of members in decisions regarding choice of treatments and specialty providers; and
• Support for the development and implementation of advanced directives.

Involvement in Decisions and Improvements at the System Level

• Inclusion of members, family and other caregivers and advocates on MCO governing and advisory boards;
• Inclusion of members, family and other caregivers and advocates on task forces and other groups organized by MCOs, State Medicaid agencies or others, to plan or track the use of managed care for people with disabilities;
• Participation of members, caregivers and advocates in the planning of quality improvement activities, including setting priorities on issues and conditions that should be the focus of such efforts; and
• Systematic use of methods to gather qualitative and quantitative information from members, caregivers and advocates to identify their priorities and their experiences using the system.

D. Resolving Member Problems and Concerns

Many, though not all, MCOs are relatively inexperienced in serving people with disabilities. For this reason if no other, it is critical to determine whether MCOs have put themselves in a position to learn quickly from their experiences and to correct their mistakes. One place where this would show is in how well they resolve the problems and concerns of particular members. In addition, as already noted, some health care needs of people with disabilities are rare or unexpected, and may only come to the surface in a crisis situation. Here are examples of indicators of whether an MCO is resolving member problems and concerns:

• Timely, courteous and helpful responses from plan customer service to member requests for information about the plan and how to use it;
• Timely, courteous and helpful responses from plan customer service to member requests to resolve questions and problems regarding access to care;
• Rapid response to resolve problems involving exceptional needs and circumstances;
• Use of appropriate procedures to hear and address member appeals, complaints and grievances (including rapid responses); and
• Reductions over time in the rate of disenrollment for cause by members with disabilities.

E. High Quality Interpersonal Interactions Between Members and Providers

Extensive consumer research indicates that everyone values highly the opportunity to have high quality interpersonal interactions with people in health care systems. There is also evidence that good patient-provider relationships have a positive effect on the management of long-term conditions and on the outcomes of health care. This domain of performance is an example of one that is important to everyone and especially important to and for people with disabilities. Here are examples of indicators in this domain:

• Proactive engagement of new members with primary care providers they have selected from the plan network;
• Providers who listen carefully to members and their caregivers;
• Providers who explain things clearly to members and their caregivers;
• Providers who treat members and their caregivers with courtesy and respect;
• Providers who spend enough time with members and their caregivers; and
• Medical office staff who are courteous, helpful and respectful of members and their caregivers.

F. Using Preventive Services to Keep Members Healthy and Functioning

This is another domain which is important to everyone and also very important for people with disabilities. Equity considerations are of special relevance here: it is critical that those with disabilities get the services we know are effective in prevention or early identification of various diseases, even though it may take special efforts to ensure they get these services at least as often as people with no disabilities. This remark applies to our first general indicator:

• Delivery of age- and gender-appropriate preventive health services.

  Considerable work has been done to identify, across the age spectrum, and by gender where relevant, specific primary and secondary prevention services and interventions of known value. Many measurement systems have identified sub-sets of these services of particular importance. However, for people with disabilities, there is another kind of prevention that is important, what public health people call tertiary prevention, i.e., the prevention of unnecessary complications and unnecessary deterioration in functioning for people who already have a defined condition. Our second general indicator is relevant to this kind of prevention. Unfortunately, less work has been done to specify and bring together services which are effective in this kind of prevention, although some have certainly been identified and are already in use as performance measures.

• Delivery of services to maintain and enhance functioning.

G. Coordinating and Integrating Medical and Non-Medical Services

Coordination of care has been consistently identified as critical for people with disabilities. One of the greatest hopes many have about managed care is that it will do a better job of coordinating and integrating care. The examples below provide different perspectives on coordination that can all be important to people with disabilities:

• Continuity of primary care and specialty providers over time;
• Ongoing coordination of medical services provided to a member by multiple providers;
• Coordination of physical and mental health/substance abuse service providers;
• Identification of non-medical services needed to maintain and enhance member health, functioning and autonomy; referral and follow-up to support delivery of such services;
• Referral to plan- and community-based psycho-social support and patient education services; and
• Ongoing coordination with providers of critical non-medical services that affect health and functioning.

H. Using State of the Art Treatments

This domain of performance comes closest to traditional quality assurance and quality improvement efforts. The domain emphasizes process measures, indicators that health care providers are using the right diagnostic and treatment procedures, medications, etc., in caring for their patients. To develop and select performance measures in this domain, it is essential to focus on a limited number of specific conditions that are sufficiently common to permit meaningful data collection and that have significant health consequences. These conditions can be identified in two different manners. The first is to build on the work already done by multi-stakeholder groups working with experts (see for example the Sentinel Conditions for Medicaid MCOs developed collaboratively by plans, State agencies and HCFA). Alternatively, the set of conditions could also be identified through multi-stakeholder interactions in a particular State. For these conditions, the strategy would then be to identify indicators that MCOs:

• Use interventions known to be of importance in the management and resolution of significant medical care problems; and
• Use interventions known to be of importance in the management and resolution of significant behavioral health problems

I. Improving the Outcomes of Care

Many believe that the most significant performance measures are those that reflect the outcomes of care, that is, whether people get better and how quickly; whether their conditions are managed effectively even if they cannot be cured; whether their functioning and quality of life are maintained to the extent possible; and whether and when they die. In addition to these traditional definitions of good outcomes, which tend to be driven by the expectations of professionals, many people believe there is another important outcome of care: the degree to which patients or plan members are satisfied with the care they receive.

In spite of the acknowledged importance of medical outcomes, project staff were consistently advised, even by those who have devoted their lives to the development of outcome measures and the conduct of outcomes research, that it would not be sensible to include medical outcome measures in our list of performance measures. Outcome measures have been used primarily in highly controlled research studies, in which an identified group of individuals are carefully tracked over time to see how they respond to a new treatment, as compared to either no treatment or a more conventional treatment. Care is taken to ensure that the groups of people who get the different kinds of treatment are extremely similar to one another. This is all to ensure that any differences in outcomes can, with confidence, be attributed to differences in the treatment.

However, these controlled circumstances do not exist when MCO performance is being measured. First, there is no guarantee that the people with disabilities who enroll in one MCO are going to be at all similar to the people with disabilities who enroll in another, or who remain in Medicaid FFS or PCCM systems. Second, there is no guarantee that these people will stay in an MCO so their progress (or lack of progress) can be tracked. Finally it is important to choose performance measures that address things the MCO can strongly influence. Many factors besides medical care are known to have a strong influence on outcomes. Research studies are carefully structured to track these other influences and take them into account, but it is very difficult and expensive to set up these controls in examining a population of MCO member.

Given these considerations, this domain should emphasize that MCOs are:

• Demonstrating improvements over time in the overall satisfaction, among members who have disabilities, with the plan, its providers, and the quality of care they are receiving.

In addition, MCOs can and should conduct their own outcome studies. Another indicator in this Domain would therefore be the following:

• With respect to conditions of special importance to persons with disabilities, the MCO conducts studies to:

  • Support or demonstrate improvements over time in the health outcomes of care;
  • Support or demonstrate improvements over time in the functional status of members; or
  • Support or demonstrate improvements over time in the quality of life of members.

IV. SPECIFIC MEASURES TO ASSESS THE PERFORMANCE OF MANAGED CARE IN SERVING PEOPLE WITH DISABILITIES

This Chapter includes measures and measurement systems that should be considered for use in assessing the performance of MCOs in serving people with disabilities. The Chapter begins by describing how measures were selected. The Chapter then presents a matrix that allows the reader to identify which major domains of measurement are addressed by each measure or measurement system. For example, if you are interested in learning about HEDIS® measures, the matrix will show which major domains of performance are, and are not, covered by these measures. After the matrix, we present, for each major domain, the specific items in the domain that are available in different measures and measurement systems. For example, if you are interested in the domain of Providing Access to Needed Services, you should consult this section to get a quick overview of the specific access measures available. Based on these summaries, we then comment briefly on which domains have extensive measures and which do not, with recommendations for measurement development.

The Chapter then moves to details about each measure. A standard set of information is presented for each measure, and these items of information are described to orient the reader. Finally, each measure or measurement system is presented.

A. How We Identified and Selected the Measures in the Guide

The measures in the Resource Guide were identified through an extensive search process that included the following:

• a review of published literature;
• a review of unpublished literature, including the proceedings and hand-out materials for relevant conferences; and
• in-depth interviews with experts in performance measurement, and in the delivery of health care services to people with disabilities.

In selecting the measures to be included in the Resource Guide, we used the following criteria:

• High relevance to the domains of performance included in our framework (see Chapter Three);
• Appropriateness as a measure of the performance of a system of care rather than for the evaluation of individual health care providers;
• Applicability to a population of people with disabilities with a broad mix of illnesses and impairments;
• Sufficient specificity for consistent measurement to occur over time and across settings;
• Evidence of scientific reliability and validity, as demonstrated either through the procedures used to develop the measure or the results of psychometric tests conducted after initial use of the measure;
• History of development or use of the measure with people with disabilities;
• Ease, cost and burden of data collection;
• Availability of the measure in the public domain;
• Availability of technical documentation for the measure and information on how data for the measure should be collected; and
• Availability of technical support for the use of the measure, preferably at low or no cost.

All the measures do not meet all these criteria. Criteria in boldface are met by all measures in the Guide. With respect to other criteria, some measures rate more highly than others, the Guide includes information that permits readers to make their own assessments on these criteria. A few measures are not available as we go to press, but are expected to be available very shortly.

B. Summary Matrix of Domains Addressed in Each Measure

Table 1 below matches each of the five measurement systems to the nine quality domains we have created. The following is a listing of the measurement systems and their abbreviations as referenced in the table:

• The Health Plan Employer Data and Information Set (version 3.0) of the National Committeee on Quality Assurance: HEDIS®
• Performance Measures for Managed Behavioral Healthcare Programs (version 2.0) of the American Managed Behavioral Health Association: PERMS
• The Consumer Assessment of Health Plans Survey: CAHPS
• The Oregon Health Plan Adult Satisfaction Survey of 1997 of the Oregon Human Resources Department: Oregon
• The Picker Institute Surveys of Adult Medical/Surgical Hospital Stays; Rehabilition Programs; and Home Care Services of the Picker Institute: Picker

C. Summary of Specific Items Available in the Measures for Each Domain

In this section, we list, for each domain in our framework, the specific items that are measured in each measure or measurement system included in the Resource Guide. Detailed descriptions of these measures and measurement systems can be found in Section F below.

Domain 1. Creating a System with the Right Capacities

HEDIS®

• Pediatric mental health network
• Chemical dependency services
• Quality assessment and improvement systems
• Provider turnover

CAHPS

• Ability to find a personal doctor or nurse

Oregon Health Plan Survey

• Ability to have a choice among different plans
• Ease of communication between personal doctor and specialists
• Creating a position of Exceptional Needs Care Coordinator (ENCC) in health plan
• Rating of helpfulness of ENCC and ENCC services

Domain 2. Providing Access to Needed Services

HEDIS®

• Mental health utilization--percentage of members receiving inpatient day/night and ambulatory services
• Mental health utilization--inpatient discharges and average length of stay
• Chemical dependency utilization--percentage of members receiving inpatient, day/night and ambulatory services

PERMS

• Telecommunications standards--standards such as a call abandonment rate of less than 5% of calls on-hold for less than 30 seconds, and an average answer time speed of five rings or fewer
• Consumer satisfaction with the time interval to the first appointment

CAHPS

• Problems getting care you and your doctor thought was needed
• Problems with delays in treatment while waiting approval from the health plan
• Access to needed help and advice from the doctor's office, by telephone
• Timely access to appointments for regular or routine care
• Timely access to a doctor or health care provider for urgently needed care
• Access to specialists
• Access to interpreters (Medicaid surveys)
• Access to special medical equipment (surveys for Medicare and people with disabilities)
• Access to physical, occupational and speech therapy (Medicare HMO survey and surveys for people with disabilities)
• Access to home health care services (Medicare HMO survey and surveys for Medicare and people with disabilities)
• Access to respite care (Medicare HMO survey and surveys for people with disabilities)
• Access to prescription medications (Medicare surveys)

Oregon Health Plan Survey

• Ease of access to a particular primary care provider
• Ease of access to specialist doctors
• Means of transportation to primary health care place
• Ease of travel to primary health care location
• Use of interpreter if needed (foreign languages, sign languages)
• Medications packaged for ease of use
• Convenient locations of pharmacies
• Ease or difficulty of early appointments, conveniently timed appointments, appointments with specialists, prescription medicines, medical advice
• Ease or difficulty of obtaining emergency medical care
• Ease or difficulty of maneuvering around medical offices (if blind, visually impaired, or use special equipment to move from place to place)
• Ease of access to counseling/mental health services
• Ease of access to alcohol or drug treatment services
• Ease of obtaining physical, occupational or speech therapy
• Ease of obtaining home health care
• Ease of acquisition of special medical equipment
• Ease of repair of special medical equipment

Picker Institute Surveys

• Timely access to hospital admission from emergency room (Hospital Survey)
• Availability of health care providers when and as frequently as needed
• Timely performance of needed tests and procedures (Hospital and Rehabilitation Surveys)
• Convenience of timing of home care visits (Home Care Survey)
• Timely availability of pain medication (Hospital Survey)
• Availability of needed special equipment in the home, and changes to home (Rehabilitation and Home Care Surveys)
• Access to sufficient number of home care visits (Home Care Survey)

Domain 3. Supporting Member Involvement in Decision Making and System Improvement

CAHPS

• Involvement in decision making about care (child surveys address involvement of both child and parent; Behavioral Health Survey may address involvement of family in decision making)
• Accuracy of information provided prior to enrollment (Medicaid surveys)
• Access to written information from plan about covered services (Medicaid surveys)

Picker Institute Surveys

• Patients have enough to say about their treatment

Domain 4. Resolving Problems and Concerns

HEDIS®

• Disenrollment rates

CAHPS

• Ability to get needed information from health plan customer service
• Resolution of complaints and appeals (Medicaid, Medicare surveys)
• Reasons for disenrollment (Disenrollee survey)

Oregon Health Plan Survey

• Notification by health plan and/or medical assistance worker of rights to complain or appeal
• Access to information from membership services
• Access to easily understandable information about benefits and services
• Knowledge about Ombudsman staff at the Office of Medical Assistance Programs (OMAP)
• Ease of access to OMAP's Ombudsman staff
• Helpfulness of information received from Ombudsman staff

Picker Institute Surveys

• Timely response to complaints (Home Care Survey)
• Agency willingness to change home care providers (Home Care Survey)

Domain 5. High Quality Interpersonal Interactions Between Members and Providers

CAHPS

• Providers listen carefully (in children's surveys, questions are asked re parent and child)
• Providers explain things clearly (in children's surveys, questions are asked regarding parent and child)
• Providers spend enough time
• Providers show respect for what patients say (in children's surveys, questions are asked regarding parent and child)
• Medical Office Staff are respectful and courteous
• Medical Office Staff are helpful
• Problems interacting with providers because of language difficulties (Medicaid surveys)
• Providers offer reassurance and support to parents regarding how they care for their child (Children with Special Needs survey)

Oregon Health Plan Survey

• Providers listen without interrupting or rushing
• Providers explain things clearly
• Providers show respect for patient
• Providers spend sufficient time with patient
• Providers follow through on test results or after care
• Health care professionals do not convey conflicting information
• Providers provide enough information about a health care condition
• Ratings of personal doctor or nurse

Picker Institute Surveys

• Providers listen carefully
• Providers explain treatments and answer questions clearly
• Staff and providers are courteous and helpful
• Home care providers are courteous to family and friends (Home Care Survey)
• Staff explain reasons for delays (Hospital Survey)
• Providers treat patients with respect and dignity
• Patients have enough to say about their treatment
• Providers explain risks and benefits of surgery and how patient will feel after surgery (Hospital Survey)
• Providers discuss patient's anxieties and fears and/or offers encouragement
• Providers inspire confidence and trust
• Providers explain test results (Hospital Survey)
• Providers talk to, involve, provide information to family members
• Providers do not talk about patient in their presence as if they were not there
• Patients get enough privacy (Rehabilitation Survey)

Domain 6. Using Preventive Services to Keep Members Healthy and Functioning

HEDIS®

• Advising smokers to quit
• Flu shots for older adults
• Cervical cancer screening
• Breast cancer screening
• Childhood immunization status
• Adolescent immunization status

CAHPS

• Advising smokers to quit (Medicare, Medicaid surveys)
• Flu shots for older adults (Medicare survey)
• Provider discussion of child development (Children with Special Needs survey)
• Provider encouragement of preventive health behaviors (Children with Special Needs survey)
• Reminders about check-ups and preventive care (Medicaid Children's surveys)

Oregon Health Plan Survey

• Provider discussion of prevention and maintenance
• Provider reminder of mammogram, pap test, quitting tobacco, cholesterol test, weight control, prostate screening, alcohol or drug screening

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

HEDIS®

• Arrangements with public health, educational and social service organizations
• Case management

CAHPS

• Availability of case management (Children with Special Needs survey)
• Linkage to schools (Children with Special Needs survey)

Oregon Health Plan Survey

• Arrangements to "watch kids" at home or provide transportation
• Patient gets help with coordination of medical care needs (from medical provider, from Exceptional Needs Care Coordinator (ENCC) at health plan, from Case Manager at social services, other)
• Rating of helpfulness of provider in giving information about other agencies or services

Picker Institute Surveys

• Providers are consistent in what they say to patients (Hospital Survey)
• Presence of one person to coordinate care provided by rehabilitation team (Rehabilitation Survey)
• Support for family involvement in providing care, including care post-discharge (Rehabilitation Survey)
• Arrangements made for post-discharge care from other agencies (Rehabilitation Survey)
• Consistency of provider across time (Home Care Survey)
• Rating of instruction given to patient to take care of medical needs at home
• Patient ratings of provider coordination (Hospital and Rehabilitation Survey)

Domain 8. State of the Art Treatments

HEDIS®

• Beta blocker treatment after a heart attack
• Eye exams for people with diabetes
• Treating children's ear infections
• Prenatal care in the first trimester
• Check-ups after delivery
• Follow-up after hospitalization for mental illness

PERMS

• Encouragement by provider to use self-help or consumer-run programs
• Medication management for individuals with schizophrenia
• Family visits for children undergoing mental health treatments
• Ambulatory follow-up within 7 and 30 days of discharge for mental health
• Ambulatory follow-up within 7 Days of discharge for substance abuse
• Ambulatory follow-up after hospitalization for major depressive disorder

Oregon Health Plan Survey

• Rating of home health care

Picker Institute Surveys

• Hospitals provide machinery patients can use to give themselves pain medication (Hospital Survey)
• Staff does all they can to help patients control pain (Hospital and Home Care Surveys)
• Patients are given information on discharge about medications and how to take them, danger signals and other follow-up care (Hospital and Rehabilitation Surveys)
• Patients are given information on nutritional needs, pain control, etc. (Hospital and Rehabilitation Surveys)

Domain 9. Improving the Outcomes of Care

HEDIS®

• Readmission for specified mental health disorders
• Readmission for chemical dependency

PERMS

• Rate of engagement with treatment for substance abuse

CAHPS

• Ratings of personal doctor
• Ratings of specialists
• Ratings of all health care providers
• Ratings of health plan

Oregon Health Plan Survey

• Patient rating of health status compared to one year previously

Picker Institute Surveys

• Patient ratings of hospital care (Hospital Survey)
• Patient ratings of rehabilitation services (Rehabilitation Survey)
• Patient ratings of home care (Home Care Survey)

D. Areas Where Measures Are and Are Not Available

The preceding summary of the specific measures available in each domain reveals clearly that in certain domains there is a rich and varied choice of measures, while in others there are relatively few measures. There is an especially wide choice of measures in these areas:

• Domain 2. Providing Access to Needed Services; and
• Domain 5. High Quality Interpersonal Interactions Between Members and Providers.

Research has indicated that these two areas are of special importance to all kinds of patients and consumers, in the context of managed care, and they are certainly of importance to people with disabilities. In addition, it is easy to gather data on these topics from patients and consumers themselves, who are often in the best position to provide useful information on these topics.

There is a reasonable number of measures available in these two areas:

• Domain 6. Using Preventive Services to Keep Members Healthy and Functioning; and
• Domain 8. State of the Art Treatments.

These two areas have been a focus of a good deal of measurement development efforts that build on the foundations of measurement of clinical quality of care. However, there is still work needed in these areas to develop measures that address conditions and concerns specific to people with disabilities, including more and better measures of mental health and developmental problems. In contrast, there are relatively few measures available in Domain 9, Improving the Outcomes of Care. As noted earlier, there are methodological difficulties in applying traditional clinical outcome measures in the assessment of the outcomes of care for a population of members in MCOs. In the early stages of the process of measuring the performance of MCOs in caring for people with disabilities, other arenas may therefore have higher priority for new measurement development.

In particular, the scarcity of measures in the following areas is of concern:

• Domain 1. Creating a System with the Right Capacities
• Domain 3. Supporting Member Involvement in Decision Making and System Improvement
• Domain 4. Resolving Problems and Concerns; and
• Domain 7. Coordinating and Integrating Medical and Non-Medical Services.

Several of these domains can and often are addressed by looking at MCO structures. Chapter Five includes several criteria sets that specify structural and procedural characteristics of MCOs that are believed to be necessary, if not sufficient, for the delivery of high quality care to people with disabilities. As we note, criteria are not, in and of themselves, measures, but the judicious use of structural criteria can help to complement other measures.

However, some aspects of these domains cannot be addressed simply in structural terms. First, we need to know if structures are not only in place, but working to meet the needs of people with disabilities. Second, some aspects of these domains go beyond structural issues. Project staff believe that it is especially critical that priority be given to measurement development in the following areas:

• the involvement of individual people with disabilities (and where appropriate their family and other caregivers) in decisions about their health care;
• the involvement of people with disabilities, their families and other caregivers, and their advocates, in decisions about the design and improvement of health care delivery systems;
• the degree to which member problems, concerns and grievances are promptly addressed and resolved to their satisfaction;
• continuity and coordination of medical care services; and
• coordination of medical and related social and ancillary services.

E. Overview of Information to be Provided About Each Measure or Measurement System

The following information is presented below, wherever it is available, for each measure and measurement system in the Resource Guide:

Name of Measure: The formal name of the measure or measurement system

The Author or Organization That Developed the Measure: This would include the key contact person and location information for where to get a complete copy of the measure and any documentation that is available for the measure.

Domains of Performance Addressed by the Measure: Many of the measures in the Resource Guide address more than one of the domains of performance we have discussed in Chapter Two, Section E. We will list each of the domains of performance to which the measure is relevant and usually give examples of specific items or indicators in each domain that are addressed in the measure.

Data Collection Strategies Required for Use of the Measure: This includes the following issues:

• What type of data collection is required?
• From whom are data collected?
• What kind of sampling or oversampling is needed?
• What if any methods are available for risk-adjustment of the data collected?

What Evidence is Available to Support the Validity and Reliability of Measure: The Guide summarizes evidence drawn from the process of developing the measure and/or from psychometric testing of the measure that indicates that it is likely to be valid and reliable.

What Are the Populations and Settings in Which the Measure Has Been or Could Be Used: Very few performance measures have been developed specifically to assess care for people with disabilities. This section provides information on the populations and settings for which the measure was originally developed, and where it has been used, and gives an assessment of the additional populations or settings where it could be used.

What Written Documentation is Available to Support Use of Measure: This section lets the reader know whether there is written documentation that can be used in implementing the measure, and the level of detail of this documentation.

What Consultation is Available to Support Use of Measure: This section indicates whether technical consultation is available, from the original developers of the measure or from others, about its use. When consultation is only available at a charge this is noted.

Limits on Use of the Measure: This final section presents any limitations on the use of the measure, or on the interpretation of results, that need to be taken into consideration in a performance measurement context.

F. Descriptions of Measures and Measuring Systems

1. Selected Measures from the Health Plan Employer Data and Information Set (HEDIS®), Version 3.0

HEDIS® is one of the most well-known performance measurement systems for MCOs. It is a measurement system that includes dozens of specific measures. We have selected, from HEDIS® 3.0, those measures that are most relevant to assessing the performance of MCOs in caring for persons with disabilities.

Name of Measurement System: HEDIS®3.0

Author/Developer: National Committee for Quality Assurance (NCQA), 2000 L Street, N.W., Suite 500, Washington, D.C. 20036, Phone (202) 955-3500

NCQA is a non-profit organization formed specifically for the purpose of assessing and improving the quality and performance of health maintenance organizations. It is governed by a Board of Directors that includes representatives from purchasers of health care, from MCOs, and from the public. NCQA began its work by developing standards for the independent accreditation of HMOs. It then moved into performance measurement, with the HEDIS® system. HEDIS® 3.0 includes measures considered relevant for the commercially insured population, for people on Medicaid, and for people on Medicare. It was developed by the NCQA Committee on Performance Measurement (CPM), whose membership mirrors that of the NCQA Board.

Domains of Performance Addressed by the Measure:

HEDIS® 3.0 measures are organized by NCQA into sub-sets which do not map with our domains. We have chosen specific measures from several sub-sets, including the following: (1) effectiveness of care; (2) health plan descriptive information; (3) health plan stability; and (4) use of services. Using our domains of performance, we will list the specific HEDIS® 3.0 measures we recommend:7

Domain 1. Creating a System with the Right Capacities

• Pediatric mental health network
• Chemical dependency services
• Quality assessment and improvement systems
• Provider turnover

Domain 2. Providing Access to Needed Services

• Mental health utilization--percentage of members receiving inpatient day/night and ambulatory services
• Mental health utilization--inpatient discharges and average length of stay
• Chemical dependency utilization--percentage of members receiving inpatient, day/night and ambulatory services

(Note: Utilization rates can only be interpreted if there is a well-grounded standard for determining what the "correct" level of utilization should be for a given population of individuals. For example, if utilization rates are compared over time or across plans, there needs to be confidence that the rate at which members in different plans need a specific service is very similar. Even in that cases, only wide disparities would generate cause for concern.)

Domain 4. Resolving Problems and Concerns

• Disenrollment rates

Domain 6. Using Preventive Services to Keep Members Healthy and Functioning

• Advising smokers to quit
• Flu shots for older adults
• Cervical cancer screening
• Breast cancer screening
• Childhood immunization status
• Adolescent immunization status

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

• Arrangements with public health, educational and social service organizations
• Case management

Domain 8. State of the Art Treatments

• Beta blocker treatment after a heart attack
• Eye exams for people with diabetes
• Treating children's ear infections
• Prenatal care in the first trimester
• Check-ups after delivery
• Follow-up after hospitalization for mental illness

Domain 9. Improving the Outcomes of Care

• Readmission for specified mental health disorders
• Readmission for chemical dependency

Data Collection Strategies Required for Use of the Measure:

Type of Data Collection

The measures