U.S. Department of Health and Human Services

Meeting the Challenge of Serving People With Disabilities: A Resource Guide for Assessing the Performance of Managed Care Organizations

Shoshanna Sofaer, Dr.P.H., Sabra F. Woolley, Ph.D., Kyle Anne Kenney, B.A., Barbara Kreling, B.A., and D. Richard Mauery, M.P.H.

Center for Health Outcomes Improvement Research, Center for Health Policy Research, George Washington University

July 1998

PDF Version (93 PDF pages)

---

TABLE OF CONTENTS

ACKNOWLEDGMENTS

ADVISORY GROUP

I. INTRODUCTION

1. The Context: Meeting the Challenge of Serving People with Disabilities in Managed Care
2. Purpose of the Guide
3. How the Guide is Organized

II. DEVELOPING SYSTEMS TO MEASURE THE PERFORMANCE OF MANAGED CARE IN SERVING PEOPLE WITH DISABILITIES

1. The Population of People With Disabilities
2. What Are the Health Care Needs of People With Disabilities?
3. Performance Measurement: What is It? How Can it be Used?
4. Selecting Performance Measures
5. What it Takes to Build a Performance Measurement System

III. DOMAINS OF MANAGED CARE PERFORMANCE IN SERVING PEOPLE WITH DISABILITIES DETAILED DISCUSSION OF DOMAINS

1. Creating a System With the Right Capacities
2. Providing Access to Needed Services
3. Supporting Member Involvement in Decision Making and System Improvement
4. Resolving Member Problems and Concerns
5. High Quality Interpersonal Interactions Between Members and Providers
6. Using Preventive Services to Keep Members Healthy and Functioning
7. Coordinating and Integrating Medical and Non-Medical Services
8. Using State of the Art Treatments
9. Improving the Outcomes of Care

IV. SPECIFIC MEASURES TO ACCESS THE PERFORMANCE OF MANAGED CARE IN SERVING PEOPLE WITH DISABILITIES

1. How We Identified and Selected the Measures in the Guide
2. Summary Matrix of Domains Addressed in Each Measure
3. Summary of Specific Items Available in the Measures for Each Domain
4. Areas Where Measures Are and Are Not Available
5. Overview of Information to be Provided About Each Measure or Measurement System
6. Descriptions of Measures and Measuring Systems

V. CRITERIA SETS FOR ASSESSING MANAGED CARE ORGANIZATIONS

1. Quality Improvement System for Managed Care (QISMC)
2. Key Indicators from the 1997 Santa Fe Summit on Behavioral Health
3. National Health Law Program (NHeLP) Advocacy Checklist for People With Disabilities

NOTES

REFERENCES AND RESOURCES

APPENDIX: COLLECTING DATA FOR A PARTICULAR PERFORMANCE MEASURE: AN EXAMPLE

ACKNOWLEDGMENTS

The preparation of this Resource Guide would have been impossible without the cooperation of dozens of people who demonstrated their commitment to ensuring the quality of care provided to people with disabilities. In particular, project staff would like to acknowledge the contributions of a group of advisors, whose names are listed below. The group's make-up mirrored our intended audiences, and therefore included staff in State Medicaid agencies and others who work with State Medicaid agencies; staff from managed care organizations; health care providers; and families and other advocates for people with disabilities. A subset of the group provided critical input at the inception of the project by helping us think through the framework for the document; the criteria that should be used in selecting measures; and how the document might best be organized to be useful to different audiences. A second, partially overlapping subset of the advisors actually reviewed the draft document in detail and provided extensive feedback. Of course, none is responsible for the final product, but their comments were both thoughtful and helpful and led to significant changes, and we hope, improvements. One organization, the Center for Health Care Strategies, went above and beyond the call of duty not only by providing input from their staff, but also by supporting Ruth Martin, a consultant to the Center. Ruth Martin made written contributions to the Resource Guide on the important topic of the resources needed to implement an effective performance measurement system.

We were also supported by many experts who had used, reviewed or developed measures and measurement systems that could be considered relevant to the health care needs of people with disabilities. These subject matter experts provided copies of their own materials, referred us to others working in the field, shared their own visions and caveats with us, and shaped our thinking significantly. We are especially indebted to the organizations whose measures, measurement systems and criteria sets are included in the Resource Guide. Not only did they provide us with their materials, but they also helped us present them accurately and completely.

Finally, we would like to acknowledge the continuous encouragement and assistance of staff of the Office of Disability, Aging and Long-Term Care Policy (DALTCP) in the Office of the Assistant Secretary for Planning and Evaluation (ASPE). Mary Harahan, Deputy to the Deputy Assistant Secretary/DALTCP, had the vision and the courage to pursue this project, the type of which is a departure from "traditional" ASPE undertakings. Gavin Kennedy, our Task Order Monitor, was consistently supportive, resourceful and patient. In the final stages of our work, we were pleased to get additional input and advice from Lisa Lang, from ASPE's Office of Health Policy; her expertise in health care quality was especially valuable.

ADVISORY GROUP

Juman Abujbara

Arizona Health Care Cost Containment System

Larry Bartlett

HSR, Inc.

Christy Beaudin

Value Behavioral Health

Allan Bergman

United Cerebral Palsy Associations, Inc.

Karen Brodsky

Center for Health Care Strategies

Carolyn Cocotas

Community Health Plan of the Rockies

Janet Freeze

Maryland Department of Health and Mental Hygiene, Division of Quality Assurance

Annette Hanson

Massachusetts Division of Medical Assistance

Tricia Leddy

Rhode Island Department of Human Services, Office of Managed Care

Trish MacTaggart

Health Care Financing Administration

Barrett Markland

HealthCare Matters, The Managed Care Training and Education Program

Karen O'Connor

Florida Medicaid Quality Assurance

Bobby Silverstein

George Washington University, Center for the Study and Advancement of Disability Policy

Jana Starr

United Cerebral Palsy Associations

Donald Umlah

Arizona Physicians' IPA, Inc.

Mark Wenneker

Boston Community Medical Alliance

I. INTRODUCTION

A. The Context: Meeting the Challenge of Serving People with Disabilities in Managed Care

More and more people covered by Medicaid are enrolling in managed care organizations (MCOs). Under voluntary Medicaid managed care, beneficiaries can choose between an MCO and more traditional Medicaid arrangements, such as fee-for-service (FFS) or the more structured primary care case management (PCCM) program. Under mandatory Medicaid managed care, beneficiaries must enroll in an MCO under contract with their State's Medicaid agency. Most State Medicaid managed care programs began by enrolling people eligible for Aid to Families with Dependent Children, or as it now called, Temporary Assistance for Needy Families (TANF). However, several States have now begun to implement, or seriously consider, the use of managed care for people covered by Medicaid because they are eligible for Supplemental Security Income (SSI); all eligibles in this category are by definition persons with a disability.

Several advantages have been attributed to Medicaid managed care. Proponents believe it will:

• enhance access to high quality health care providers who have previously been unwilling to serve Medicaid eligibles;
• do better at integrating and coordinating care over time and across service delivery settings than the more decentralized FFS system;
• avoid unnecessary and costly care;
• reduce the inappropriate use of service sites, such as hospital emergency rooms, that are often used by low-income people with limited access to primary care physicians; and
• focus more on the provision of preventive health services.

However, others have concerns about whether MCOs can and will meet the health care needs of Medicaid eligibles. They worry, for example, that:

• MCO networks will not include traditional providers of care to Medicaid eligibles (e.g., public hospitals and clinics; community-based health centers and clinics; and inner-city teaching hospitals) who know the special health care needs and personal characteristics of Medicaid eligibles, and are more able and willing to meet those needs; and that
• the financial incentives of capitated payments will cause MCOs to place barriers to access to needed health services.

These concerns become more significant when decisions are being made about the use of MCOs to care for people with disabilities, many of whom need a complex (and sometimes uncommon) mix of health and social services to maintain health and functioning. One advantage of MCOs is that it is easier to measure, and thus to improve, the performance of a given MCO than it has been to measure, and especially to improve, the performance of a community's FFS system. The expansion of managed care has been accompanied by increased attention to performance measurement for health care delivery systems, and an increased emphasis on the need to hold systems accountable for their performance.1

While there is little evidence that Medicaid FFS as a system took very good care of the population of persons with disabilities, it is clear that specific health professionals, facilities and programs have provided excellent and responsive care to particular patients in particular communities. For many, the ultimate value of managed care will be determined by whether it works well for people with complex and specialized health care problems and needs, the kind of problems and need of many persons with disabilities. As more and more people with disabilities enroll in managed care, it is critical, for us all, to determine whether MCOs are performing well in meeting their needs.

Comprehensive systems for measuring the performance of health care systems in caring for persons with disabilities do not yet exist. At the same time, thousands of individual quality measures exist but it is often difficult to discern which will be most reliable and relevant to measure MCO performance in caring for people with disabilities. In these circumstances, it is easy to respond either by (1) doing little or nothing to measure MCO performance or (2) mounting costly efforts to measure hundreds of highly specific aspects of quality that may fail to provide a coherent picture of performance. This Resource Guide is designed to help those who want to begin to work toward a comprehensive system, today, by using measures available right now that have a clear relationship to domains of performance important to the care of persons with disabilities.

B. Purpose of the Guide

This Resource Guide is designed to support efforts to measure and improve Medicaid MCO performance for people with disabilities. It presents the results of an extensive search for existing performance measures that can be used for quality measurement and improvement by:

• State Medicaid agencies;
• managed care organizations themselves;
• providers within MCOs; and
• people who advocate for the health care needs of those with disabilities.

The purpose of the Resource Guide is threefold:

1. To provide an overview of the issues to be considered in measuring the performance of MCO in taking care of people with disabilities;

2. To bring together in one volume key information about measures that now exist, or are about to be made available, that can be counted on to support efforts to measure and improve the care provided by MCOs to people with disabilities; and

3. To make clear the areas where technically strong measures do not exist, in order to encourage further measurement development in these areas.

The Resource Guide is the result of a ten-month effort by the Center for Health Outcomes Improvement Research2 at the George Washington University Medical Center. This effort was supported by the Office of Disability, Aging and Long-Term Care Policy of the Office of the Assistant Secretary for Policy and Evaluation (ASPE) at the U.S. Department of Health and Human Services. ASPE commissioned the Resource Guide in response to numerous requests from the field to disseminate, sooner rather than later, the best available measures of performance in caring for people with disabilities.

Two important features of the project must be noted. First, the project was not designed to develop new measures. Rather, it identified existing measures, or measures that are about to be released. The time and resources available were not sufficient for the development of new measures. It is relatively easy to identify a dimension of performance for MCOs, and even to reach consensus across stakeholders that this dimension is significant to meeting the needs of people with disabilities. It is far more difficult, and takes considerably longer, to develop a reliable and valid way of measuring a dimension of performance. The project has helped to identify the gaps in existing performance measures, i.e. the dimensions of performance that many people think are critical but for which valid and usable measures are not available. It can, therefore, serve as a guide to future measurement development efforts.

A second feature of the project is that it did not focus on measures that would be used to assess the care provided to people, such as the frail elderly, who are in nursing homes or other residential long-term care facilities. Again, given time and resource limitations, the project's goal was to concentrate on care delivered to other populations of Medicaid eligibles with disabilities who are being enrolled in MCOs.

C. How the Guide is Organized

Following this Introduction, the Resource Guide has four chapters. Chapter Two: Developing Systems to Measure the Performance of Managed Care in Serving People With Disabilities, discusses issues that should be considered in the selection and use of performance measures, including:

• the characteristics of people with disabilities;
• the health care needs of people with disabilities;
• what performance measurement is, who can use performance measurement and how; and
• what it takes to build an effective performance measurement system.

Chapter Three: Domains of Managed Care Performance in Serving People with Disabilities, presents a framework for categorizing the aspects of MCO performance considered important in providing quality care to people with disabilities. This Chapter should be read by everyone. The framework it presents performs two roles:

1. It presents the full range of topics that are important to address with respect to serving people with disabilities.

2. It serves as an organizing tool for describing and categorizing particular measures; the framework will be used for that purpose throughout the Resource Guide.

In Chapter Four: Specific Measures to Assess the Performance of Managed Care in Serving People with Disabilities, presents measures and measurement systems that deserve serious consideration for inclusion in a performance measurement system. We are not recommending that anyone use ALL these measures. Rather, they are a good range from which to select a sub-set for implementation. This Chapter includes:

• a discussion of how we selected the measures;
• a specification of the information that will be provided about each measure or measurement system;
• a matrix that indicates what domains are addressed in each measure or measurement system in the Resource Guide;
• a summary, for each domain, of the specific measures available; and
• a detailed description of each measure or measurement system.

The Resource Guide does not include copies of actual measures, such as complete surveys, or the full technical specification of measures. The inclusion of such material would make the document large and cumbersome. Rather, the Guide provides sufficient information on each measure to help potential users decide whether further consideration of the measure is warranted, as well as information about where and how to get actual instruments, technical specifications, other documentation and in several cases technical support.

Chapter Five: Criteria Sets for Assessing Managed Care Organizations, presents material that may also be of use in examining the performance of MCOs in serving people with disabilities. Several groups have identified specific characteristics of MCOs that they think have important consequences for the care of people with disabilities. These are criteria sets, rather than formal technical measures. That is, no method has been specified for collecting information to determine whether a characteristic is present or absent. In some cases, determining whether a characteristic is present may be relatively easy; in other cases, it may be quite difficult. We have included three criteria sets in the Resource Guide because:

• unlike most technical measures, they have been developed specifically to address the concerns of people with disabilities;
• they can often be used to examine MCOs prior to finalizing contracts; and
• they provide a starting point for the development of more formal measures and data collection efforts.

Finally, a list of References and Resources provides information on other resources and documents that you might find useful in developing a performance measurement system.

II. DEVELOPING SYSTEMS TO MEASURE THE PERFORMANCE OF MANAGED CARE IN SERVING PEOPLE WITH DISABILITIES

A. The Population of People With Disabilities

To determine how best to assess the performance of MCOs in serving Medicaid eligibles with disabilities, it is important to understand key characteristics of this population. When the last U.S. Census was conducted in 1991-1992, there were 48.9 million Americans with a disability, or nearly 19.4 percent of the total U.S. population (McNeil, 1993). At that time, approximately 19 percent of persons with disabilities aged 15 to 64 were covered by Medicaid.

In 1995, according to the Health Care Financing Administration (HCFA), approximately six million people with disabilities were covered by Medicaid, of whom 1.3 million were children. Within Medicaid, those with SSI were the second fastest growing eligibility group between 1990 and 1994, increasing at an average annual rate of 10 percent (Davis and O'Brien). These authors also report that as of 1995, the disabled "make up only 15 percent of all Medicaid users, while their spending accounts for 39 percent of all program payments."

The 1994 National Health Interview Survey (NHIS) included a supplement designed to learn more about people with disabilities. Utilization of medical care among the Medicaid SSI population was slightly higher than utilization in the general Medicaid population, but similar to utilization for all people with disabilities. The survey reveals that many children with disabilities are eligible for Medicaid not under the SSI, but under TANF. This implies that measuring the performance of MCOs in serving people with disabilities, and especially children with disabilities, may be a significant concern even when a Medicaid program does not enroll SSI-eligible persons in MCOs.

Analysis of the NHIS also reveals that estimates of the size of the population with disabilities vary widely, depending upon the definition used. Some respondents who would be classified as having a disability under certain definitions (e.g., impairments, disease conditions, inability to perform certain activities of daily living) do not self-identify as having a disability. This may reflect differences in attitude between people with disabilities and their health care providers. As Harahan noted in a discussion of these results, people with disabilities "may demand control over their own lives and maximum choice."

B. What Are the Health Care Needs of People With Disabilities?

As already noted, the extent and nature of health services needed by people with disabilities varies widely. This section presents the health care needs of three subgroups: healthy people with disabilities; people with disabilities who have ongoing but not particularly complex health conditions; and people with disabilities who have complex and uncommon health conditions. Through this discussion, we highlight the issues that must be addressed in measuring the performance of MCOs in caring for a population of people with disabilities.

Taking Care of Healthy People with Disabilities

People with disabilities need the same set of preventive and curative health services as those without disabilities. Indeed, as in the general population, many people with disabilities need regular preventive care but only episodic curative care, since they are basically in good health.

In providing health services even to healthy people with disabilities, however, delivery systems must identify, and take into account, the particular challenges facing each person. For example, HCFA data cited by Davis and O'Brien (1996) reveal that "Mental impairments, including mental illness, mental retardation, and developmental disabilities, predominate among both the adult and child SSI populations." People with these conditions comprise 67 percent of adult and 57 percent of child SSI recipients. When preventive and curative medical care is being provided to this group, it must be tailored to take into account their psychological, social and developmental needs and problems. This implies two special needs even for the physically healthy person with a mental or developmental disability:

• the need for coordination of care provided by physical and behavioral health care providers;
• the need for health professionals who do not specialize in behavioral health to understand how mental health and developmental problems can affect the prevention, diagnosis, treatment and follow-up of other medical conditions.

A wide variety of other diagnoses and conditions can result in a disability that makes one eligible for SSI. Medicaid SSI eligibles also include people who have sensory impairments; neurological problems; mild to very severe limitations in their motor functioning; and respiratory limitations. As in the case of those with mental impairments, the physical, psychological and other challenges presented by their particular condition must be taken into consideration in ensuring access to medical care and in ensuring care will be appropriate to the individual. Access and appropriateness are concerns for everyone. Issues that are often of special concern to persons with disabilities include:

• facilities and equipment that do not present physical and architectural access barriers;
• specialized transportation to care sites;
• geographic proximity of health care facilities, with in-home care needed in some cases;
• alternative methods of communicating and interacting with enrollees that are appropriate to those with sensory or linguistic impairments; and
• active participation of the individual people with disabilities and where appropriate his/her family or other caregivers in decisions about their care.

Ongoing Health Care for Persons with Disabilities

Other persons with disabilities resemble (indeed may be) persons with chronic medical conditions. That is, in addition to regular preventive services and episodic curative care, they need specific services on a regular basis for an ongoing condition. They (and often their family) must often participate actively to make sure their condition is well managed. Since their condition may never be "cured," the focus of attention shifts to their ability to live as full and active a life as possible is maintained, and to avoid preventable deterioration and complications of their condition. For this group of persons with disabilities, the following needs become especially important:

• access to a regular health care provider (whether a primary care provider or a specialist) who understands their particular problem and how it is likely to affect their life;
• ability to maintain a relationship with this regular health care provider over time;
• careful tracking by the health care system of the individual's condition, of their response to treatment, of changes in their life circumstances and daily functioning; and of the emergence of other medical problems which can affect, or be affected by, their ongoing condition;
• access to patient education, nutrition, exercise and other services that support effective management of the underlying condition;
• access to services needed to prevent complications or deterioration or to treat them promptly when they do occur;
• access to health services that can maintain or enhance the person's independence and improved functioning in daily life;
• links to non-medical services (e.g., housing, education, employment training and placement, transportation, etc.) that also support independence and improved functioning in daily life;
• active involvement of the person and where appropriate his/her family in decisions about treatment alternatives and in the implementation of treatment plans; and
• support for family members who care for or are simply significantly affected by the health and functioning of a person with an ongoing and disabling condition.

Caring for People with Complex and Uncommon Needs

Compared to the general population, a higher proportion of people with disabilities face quite serious and often uncommon medical problems, that must be treated or managed effectively, and that can significantly influence the treatment of other medical problems that occur over their lifespan. Their health care needs are complex and specialized. Multiple providers may care for an individual patient at any given point in time. The specific services required may change over time. Given these circumstances, the following issues, in addition to those already mentioned, become particularly important:

• access to highly specialized providers and facilities experienced in diagnosing and treating the individual's condition;
• access to rarely performed tests and procedures;
• access to rarely prescribed medications, medical supplies and equipment;
• coordination of care across providers;
• continuity of care over time; and
• careful tracking of changing circumstances and needs as well as relevant medical advances.

Most MCOs are not organized, at the outset, to take care of the full range of complex and uncommon medical problems that may be faced by their members. Their network may not include the required specialists. Their formulary may not include the required medications. MCOs that agree to take responsibility for a population of persons with disabilities need to expect the unexpected, and be ready, willing and able to respond. The responsiveness of MCOs to complex and uncommon medical problems is influenced by the following factors:

• the use of a reasonable definition of medical necessity that does not exclude services essential to diagnosing and treating uncommon conditions;
• the willingness and ability to add highly specialized providers and facilities to their network in order to diagnose and treat uncommon conditions;
• the willingness and ability to provide medications not included on their formulary when essential to treating uncommon conditions or preventing complications;
• the availability of user-friendly, well-understood methods that members can use to appeal denials of care and get prompt responses; and
• the availability of care coordination services.

C. Performance Measurement: What is It? How Can it be Used?

This section of the Resource Guide addresses how to think about and plan for a system for measuring MCO performance in serving people with disabilities. It begins by defining performance measurement, and then describes how performance measurement can be used by different groups, and what resources are needed to use it well. The section ends with a discussion of how to enhance, over time, our efforts to measure and improve performance.

What is Performance Measurement?

In this Guide, performance measurement is defined as:

the process of using formal, scientifically grounded tools and methods to collect information about a health care delivery system, such as an MCO, to determine whether its characteristics and actions, and the consequences of its actions, meet expectations.

Performance measurement is a key element in holding health care delivery systems accountable for what they do. It is intended to inform decisions and guide actions. It is not measurement simply for the sake of measurement, or to conduct research.

As implied in our definition, performance measurement involves comparing performance against a set of expectations about what an MCO can and should be doing for its members. These expectations come in part from what research tells us can and should be done to achieve desired outcomes. However, the performance measures we choose, and in particular what we choose to measure, invariably reflect, in addition, values and preferences.

This leads to a question: whose expectations, and whose values and preferences, should drive the development of performance measures? The development of effective performance measurement systems will require participation from many different stakeholders: purchasers (such as State Medicaid agencies); policy makers; MCOs; health and social service providers; and consumers, including persons with disabilities and their families and other informal caregivers. Each of these stakeholders is likely to have distinct values, preferences and expectations. To develop a performance measurement program, it is essential to develop sufficient consensus about expectations. While it is difficult to reach complete agreement on expectations, it is clearly possible to identify a set of important and widely shared expectations.

How Can Performance Measurement be Used?

Performance measurement can be undertaken by different organizations, for different purposes. We will discuss how measuring performance of MCOs in serving people with disabilities might be used by four different and significant constituencies: State Medicaid agencies; MCOs; people with disabilities and those who care or advocate for them; and health care providers.

State Medicaid agencies can use performance measurement to:

• identify areas where improvement is needed in the performance of particular MCOs, set goals for improvement and in some cases incorporate them into contract specifications;
• identify exemplary behavior in particular MCOs to highlight and disseminate, or use as a benchmark for the performance of other plans;
• provide useful and reliable comparative information to Medicaid eligibles who are trying to choose among various MCOs they can join;
• select an MCO for persons who do not explicitly choose one for themselves;
• decide whether to contract with a given MCO in the future;
• determine if there should be a ceiling set for new enrollments in a given MCO;
• in extreme cases, introduce intermediate sanctions or even terminate a contract with a plan; and
• identify patterns of poor performance across multiple plans that may signal the need for broader changes in policies, procedures or even legislation.3

Managed care organizations can use performance measurement to:

• identify where they need to focus internal quality improvement activities;
• track their progress over time in making quality improvements;
• make decisions about which provider groups or facilities it wants to include in its network in the future;4 and
• market themselves to potential enrollees.

People with disabilities, their caregivers and advocates can use performance measurement to:

• make more informed choices both between managed care and FFS systems and among various MCOs;
• become more capable of making their own choice, instead of being arbitrarily assigned to an MCO by the State Medicaid agency;
• identify and publicize significant problems either in a particular MCO or across several MCOs, in a way that moves beyond individual anecdotes;
• identify and publicize positive findings to highlight what can be achieved for people with disabilities, how it can be done, and who is doing it.

Health care providers (such as hospitals, health centers, individual physicians and physician groups) can use performance measurement to:

• support their own quality improvement efforts;
• highlight for the public the things they do best; and
• decide which MCO networks they would like to join or remain a part of.

Performance Measurement and Comparisons

Performance measurement always involves making comparisons. Here are some comparisons that can be made using performance measurement:

• comparing different MCOs to each other:

  who is performing better at ensuring that patients are actively involved in decisions about their care? who is not doing as well?

• comparing each MCO's performance to an explicit standard:

  do people get seen for urgent care within 12 hours? do at least 90 percent of children get all required immunizations by the age of two? are sign language interpreters always available for regularly scheduled appointments with hearing impaired persons?

• comparing how an MCO performs at a certain point in time to how they performed at another point in time:

  is the MCO reducing how long patients wait in the office before they are seen? or are patients actually waiting longer?

• comparing the performance of MCOs in caring for people with disabilities to their performance in caring for other members:

  how do rates of mammography compare between women over 50 with and without disabilities? how often do people with disabilities disenroll from a given MCO because of quality concerns, as compared to disenrollment among those without disabilities?

• comparing the performance of MCOs, across the board, with the performance of other delivery systems such as FFS or PCCM:

  how do people with disabilities who join MCOs rate their access to specialists, as compared to those who remain in FFS or PCCM?

Different kinds of comparisons reflect, sometimes implicitly and sometimes explicitly, different expectations. For example, when MCOs are being compared to each other, they are implicitly measuring their performance against the "average" performance in the group. Saying that a particular MCO is "better than average" may be misleading if everyone is performing poorly. Saying performance is "just about average" may similarly be misleading is everyone is performing very well.

On the other hand, comparing each MCO's performance to a standard makes expectations very explicit. Standards, like measures, have to be developed. They can be derived from:

• what can actually be achieved because at least some pathbreaking MCOs have achieved it (this is often called a benchmark):

  for example, a small group of MCOs has found methods to ensure that primary care physicians diagnose depression and make appropriate referrals where needed for 80 percent of their patients

• what research shows is essential, or critical, to achieve desired outcomes:

  for example, when 90 percent of a population has been immunized, the population as a whole achieves "herd immunity" reducing to virtually nothing the chance of disease transmission

• what stakeholders agree is both achievable and morally right:

  for example, a standard can be set that no one should face physical barriers to access to health care facilities

Finally, when the performance of MCOs in serving people with disabilities is compared to their performance in serving other members, there is an implicit expectation that people with disabilities deserve, can, and should get at least the same quality of care as everyone else.

D. Selecting Performance Measures

There are innumerable attributes of an MCO that can affect the health, functioning, quality of life and satisfaction of the people it serves. There are thousands of measures of the quality of medical services, most designed to determine whether, for a patient with a particular diagnosis or condition, the correct services were provided at the right time by the appropriate people and achieved the desired consequences. But very few MCO attributes, and very few condition-specific quality measures, are likely to work well as part of a system of performance measurement. Why? For one thing, no one has the resources to measure everything that could be measured. No one has the time or resources to use all the information that would be generated by such a massive measurement effort. Indeed, it is important to avoid spending so many resources on performance measurement that it detracts from, rather than adds to, our ability to provide good service to people with disabilities.

To build a performance measurement system, it is critical to select a limited number of good measures that together provide a coherent picture of a health care delivery system, rather than measure everything. As noted earlier, the performance measures selected will get the attention of MCOs, so it is important to select the right set. Here are some criteria that can be used to select measures; they were also used to select the measures included in this Resource Guide:

1. The measure should tell you about something that is considered of great significance.

   As noted above, performance measures reflect expectations and values. Some things that an MCO does (or does not) do are far more significant that others, depending on your expectations and values.

2. The measure should address an aspect of the performance that the MCO can significantly influence.

   Performance measurement is a tool to hold MCOs accountable. It is neither sensible nor fair to hold an organization accountable for what it cannot influence. Notice this criterion uses the term influence rather than control. No organization can control anything completely. However, MCOs have significant influence over many things they do not control. Another way to put this is this: does the measure address something that is actionable, something that the MCO can work to improve? does it give the MCO clear direction about what it needs to work on?

   Related to this criterion is the issue of what the MCO is contractually required to provide to its members, and in particular to its members with disabilities. Thus, for example, if a contract between a State Medicaid agency and an MCO does not specify that a given service must be provided to enrollees, it may not be appropriate to measure whether or not that service is indeed provided.

3. The measure should tell you something that reflects the performance of an MCO as a system of care, rather than a very narrow aspect of its functioning.

   This criterion is somewhat difficult to understand and apply. An example may help. When health care delivery systems in different countries are compared, one of the performance measures that is almost always used is the infant mortality rate. This measure meets our first two criteria well: increasing the number of newborns who survive to at least the age of one is clearly important; and while the health care delivery system cannot control all the factors that influence infant mortality, it has a significant influence on many others. It also meets this third criteria, because reducing infant mortality requires that many different parts of the health care system do a good job. The infant mortality rate reflects whether women get prompt access to pre-natal care; whether the care is consistent and effective; whether the most up-to-date tests and treatments are being used; whether potentially high-risk pregnancies are being identified early; and whether action is being taken to reduce or address risks swiftly and effectively.

   Another way to think about this criterion is in terms of whether a particular action that an MCO can take would have a big effect on the entire experience a member or a patient has in the MCO. For example, whether a new member of an MCO is linked to a primary care provider quickly (say within 30 days) could be critical for their ability to use and benefit from the entire system. Another example of special importance to people with disabilities is whether the MCO has an efficient and timely process for bringing new, highly specialized providers into their network when a new member enrolls who needs such providers to maintain or improve health and functioning.

4. The measure should let you make the comparisons you want to make.

   As discussed above, performance measurement always involves comparisons. Measures should be selected that are relevant to the comparisons being made. Just as important, measures have to be selected that can be implemented across all the organizations or systems being compared. In particular, the data needed for the measure has to be available across all these organizations and the measures need to be sufficiently specific to ensure that data are collected exactly the same way.

5. The measure should address an event that can be observed often enough to produce reliable results.

   Most performance measures are rates; they measure how often, or for what proportion of people, an event takes place. The denominator of a rate indicates the maximum number of times, or the maximum number of people, for a given event. For example, the denominator for a disenrollment rate would be the number of people enrolled in a plan over a given time period. The numerator of a rate indicates the actual period of time, or the actual number of people, for a given event. Thus, the numerator for a disenrollment rate would be the number of people in a plan who disenrolled (typically for specific reasons related to their dissatisfaction with care) in that same time period. When the denominator or the numerator are likely to be quite small, it becomes difficult, for statistical reasons, to have confidence in the reliability of a rate. It also becomes very difficult to make comparisons.

   This "small number problem" makes it difficult to include as performance measures events that are specific to a particular health problem or condition, especially if that condition is not very common in the general population, in the population of people with disabilities, or in the population of people enrolled in a particular MCO or group of MCOs. For this reason, we have emphasized, in the Resource Guide, measures that are applicable to larger groups of people, rather than condition-specific measures.

E. What it Takes to Build a Performance Measurement System

Measures are not the only resource needed to build, and to use well, a system of performance measurement. In this part of the Resource Guide we will discuss the other resources that need to be in place to build a system.

Planning and Designing a System of Performance Measurement

As noted above, performance measurement requires that we clarify expectations. Also as noted, many constituencies have an important stake in what gets measured, how it is measured, how it gets reported, and to whom. For this reason, it is highly desirable to design systems of performance measurement with input from multiple stakeholders. In most but not all cases, the lead in development of such systems will come from State Medicaid agencies. The lead organization should convene all stakeholders, including other State agencies who provide health and related social services to people with disabilities. Experiences in several States indicates that the initial planning and design process can take up to a year. The convenor needs to have skills in planning, in facilitating group processes and in managing conflict, as well as technical knowledge. Some may find it useful to contract with an external consultant as a facilitator, since an outsider may appear more neutral than any of the parties.

Resources Needed to Collect and Analyze Performance Data

No matter how carefully measures are selected to assess MCO performance in caring for people with disabilities, the system will fail unless accurate and comparable data are collected in a systematic manner and unless it is analyzed and interpreted correctly. Two key players, State Medicaid agencies and MCOs, are most likely to be directly involved in data collection and analysis. Two kinds of data are likely to be collected for performance measures: primary data, collected for example through surveys of members; and secondary data, collected from existing administrative and clinical records and information systems.

State Medicaid Agency

Within State Medicaid agencies, a critical resource is people. One or more dedicated staff members will be needed with knowledge of the following:

• performance and quality measurement;
• the strengths and weaknesses of different approaches to data collection;
• basics of survey research and sampling;
• contracting and coordinating with third-party vendors to conduct surveys;
• drawing survey samples using State and MCO records;
• the structure, potential and limitations of State and MCO management and clinical information systems;
• methods for auditing and validating data provided by MCOs;
• data tracking and management;
• alternate approaches to the transformation of raw data into rates and other findings;
• methods for risk adjustment of performance and quality data;
• comparative analysis of data; and
• presentation of data to multiple audiences.

States need to decide how they want to receive performance information. There are three basic choices:

1. MCOs submit the calculated performance measure;

2. MCOs submit the data required to calculate the numerator and denominator of the performance measure and the State calculates the measures;

3. for measures calculated from administrative data only, the MCOs submit all claims or encounter data and the State calculates the measures.

The approach chosen has implications for the extent and nature of computer hardware and software resources, and for how many and what types of staff are needed to receive, clean, and load the data and calculate the performance measures.

States must also choose how they will audit and validate data. The need for validation increases as the MCO, rather than the State, takes responsibility for data collection and the calculation of performance measures. But much of the data will inevitably be based on MCO records. Unless the data are audited, the State cannot assess the level of confidence that it should have in the reported performance measures or correct the problems with the performance measures. There are several ways in which the State can audit the data:

• using State staff;
• using a contractor, such as the External Quality Review Organization (EQRO); or
• requiring that the MCOs hire independent auditors to certify their performance measures.5

Managed Care Organizations

To participate effectively in a performance measurement system, MCOs need three broad categories of resources:

• staff with appropriate knowledge and analytic skills;
• an appropriate management/medical information system; and
• staff with appropriate skills to collect data from clinical records.

Specific staff expertise includes:

• performance and quality measurement;
• how to translate the definition of a performance measure into detailed operational specifications for data collection;
• an understanding of MCO program operations, including related quality assurance and quality improvement activities;
• and understanding of the potential and limitations of the MCOs' management and clinical information systems (and the quality of information systems at the provider level);
• the history of data collection and information systems to identify changes that would affect the reliability of trend data;
• data collection methods, including abstraction of data from non-electronic medical records;
• data tracking, cleaning and management; and
• auditing and validating methods.

If MCOs choose to implement performance measures based on surveys, they will also need either an in-house survey capacity or more likely the capacity to select and work with an outside vendor.

As performance measurement and reporting have become more critical for MCOs, the need for management information systems that can support these efforts has grown. In an ideal world, all medical records would be automated and MCOs would have direct access to clinical data that could be combined with administrative data, such as enrollment records, to produce timely, accurate performance measures. Unfortunately, current reality is far from ideal. Therefore, clinical information is derived, to the extent possible, from claims and/or encounter data. Information is further limited by the type of coding used for office visits and procedures and the coding used for diagnosis. For Medicaid members, enrollment data come directly from the State and are not under the control of the MCO; these data are sometimes critical to specifying the population for which a measure will be relevant. Therefore, collaboration at the operational level will almost always be needed between the State Medicaid agency and MCOs in order to generate many performance measures.

An MCO's ability to participate in an effective performance measurement system depends on its having a management information system (and related staff) that:

• Collects and reconciles enrollment data and maintains an enrollment history;
• Collect and processes claims/encounter data in a timely and accurate fashion and maintains a claims history;
• Can integrate data from other sources, such as pharmacy claims data from a third-party vendor; and
• Has a flexible report-generator or can export data to a report generation program in order to calculate the performance measures.

Many performance measures will require the use of clinical information (that may well not be computerized) as well as administrative data (that is more likely to be computerized).6 This means it is important to have staff with expertise and experience in the abstraction of medical records. As important is a healthy relationship between the MCO and its providers. Especially in more loosely organized and decentralized provider networks, the MCO will need to work carefully with its providers to ensure that data are reliably and consistently collected. In this context, auditing and validation are even more important.

Level of Resources

The amount and type of data collection and analysis resources needed by States and MCOs depend on the number and type of performance measures chosen, the frequency of reporting, and the level of statistical confidence needed. In general:

• More frequent reporting requires more resources, unless data are readily available from a computerized management information system.
• Measures that require manual data collection (e.g., medical records review) require more resources than measures that can be calculated directly from computerized records.
• When sampling is used to collect data to calculate measures (for example with manual medical records review or surveys), higher statistical confidence levels require more resources than lower confidence levels.
• If different samples have to be drawn for multiple measures, more resources are required. This can be somewhat mitigated when data for several measures can be collected from a single medical record.
• When oversampling or screening is used to ensure that a sufficient number of people with disabilities are included, higher levels of resources are needed.

Resources Needed for Disseminating Performance Information

All too often, based on traditional practice, performance measurement is viewed as of interest only to technical professionals. In fact, however, a critical value of performance measurement is that it promotes both general accountability to the public and the generation of information that can support decisions and actions made by individual members of the public. Performance data can only be used by those who have access to it. In planning a performance measurement system, attention needs to be given to whether, how, when and to whom performance data will be disseminated.

MCOs and State Medicaid agencies may find themselves uncomfortable at the thought that they will not be the only people who see, interpret and use performance information. It does "raise the stakes" both on the relevance and quality of the data when wider dissemination is planned. In some ways, however, this a good reason to pursue dissemination: the quality of data may rise when people know that the public will see it.

The implication is not only that dissemination needs to be considered in planning, but also that resources will be required to develop and implement the dissemination strategy. The resources include staff or third-party consultants and vendors with skills and experience in:

• communication, including writing and graphic design;
• the use of formal and informal media and channels for reaching target audiences; and
• tailoring the presentation of data to the characteristics and interests of the audience, including their literacy levels, cognitive and sensory impairments, language preferences, and trusted information channels.

People with disabilities, their caregivers and advocates are not just another audience in this context; they understand best how they can be reached effectively and it is wise to involve them from the outset in planning dissemination efforts.

Enhancing Efforts to Measure and Improve Performance

At the outset, State Medicaid agencies and MCOs will not have all the resources needed to implement an ideal performance measurement system, including in particular:

• skilled and knowledgeable staff;
• effective systems for collecting and analyzing data;
• methods for disseminating the results of performance measurement; and
• an accepted forum where various stakeholders can participate in the process of planning and trouble-shooting.

Performance measurement systems will be "works in progress" for many years. They will not be ideal and should not be expected to be ideal at the outset. This is one area where the adage that "the perfect can be the enemy of the good" is very applicable.

The development of performance measurement systems has to start somewhere. We strongly recommend beginning with a limited set of measures that are scientifically well-grounded, relevant to the concerns of stakeholders, and feasible to implement given the current state of available data collection and information systems. The planning process should be designed to help you identify these measures. It will also serve to identify the more basic structural barriers to using performance measurement. Often, simultaneous work may be needed on structural barriers and the first stages of building a performance measurement system.

Experience can be a harsh but excellent teacher. For this reason, we also recommend learning about and staying in touch with others who are embarked on similar efforts, so you can gain from their experience as well as your own. The reverse is true: share your experiences with others.

The long-range development of performance measurement systems clearly requires investment of resources. Acquiring and maintaining those resources also means that the benefits of performance measurement be made clear to those with influence over resource allocation. Ultimately, however, development of an effective performance measurement system will require that senior officials in State Medicaid agencies, MCOs, in the community of people with disabilities, and among health care providers, make a commitment to using objective information to assess and improve our systems of care.

III. DOMAINS OF MANAGED CARE PERFORMANCE IN SERVING PEOPLE WITH DISABILITIES

This Chapter presents a framework of domains of managed care performance that the project identified as critical to serving people with disabilities well. These domains permit measures to be categorized into broad topics that have been identified as important to serving the needs of people with disabilities. These domains and the example measures we present in each were based on:

• interviews with numerous experts and stakeholders;
• review of research on what people find important about their health plans; and
• analysis of many documents (see References and Resources).

The framework represents a synthesis, by project staff, of many sources of information, including existing approaches to categorizing measures. For example, quality measures were first categorized by Donabedian into three types: structure, process and outcome. Many people are familiar with this framework, but it does not address the actual content and substantive issues addressed by measures. It is therefore less useful for those trying to think about performance measurement and to select specific measures.

This framework of domains represents an ideal: it would be highly desirable to find well-grounded and usable performance measures across all the domains. However, our search for measures indicates that some domains are thinly populated compared to others. Nevertheless, it is useful to have a comprehensive set of issues and concerns to consider, even if measures do not exist today. By starting with a comprehensive set of domains, it has been possible to identify clearly the gaps in existing measures, thus providing direction for future measurement development efforts.

The nine major domains in our framework are presented first. Details and discussion of each domain follow.

Detailed Discussion of Domains

A. Creating a System with the Right Capacities

As noted above, quality measurements have been distinguished in terms of structure, process and outcome. The earliest quality measures emphasized structure and were typically used to give a facility a license or to accredit a program. This first domain of MCO performance also emphasizes structure, but we use here the broader term capacity. One great advantage of structural or capacity measures is that they can be used prospectively, that is before people with disabilities are enrolled in a particular MCO. They can also be used even after enrollment, to make sure that capacities are in fact present and perhaps being enhanced. Here are examples of indicators of the capacity of an MCO to take care of people with disabilities. As with all the lists in this section of the Resource Guide, these examples are not considered to be complete; you may well identify additional indicators and measures, as we have.

• The number, mix and qualifications of providers included in the MCO network;
• The stability of the MCO provider network;
• The presence of procedures for adding new providers to meet unexpected needs of members;
• The presence of well-designed and active quality assurance and quality improvement systems;
• The presence of a system for assessing and tracking the health and functional status of members (especially those with disabilities); and
• The presence of a method for coordinating the care provided to members (especially those with disabilities) with complex or exceptional needs.

Note that in these examples, the emphasis is on whether the capacity is in place. We view this as a necessary, but not sufficient, condition for high performance. It is also important to determine whether these capacities are actually used and whether or not they are effective. These issues are addressed in other domains.

B. Providing Access to Needed Services

Access to care has long been a critical element of any assessment of a health care delivery system. When access barriers exist, people have less chance to get the services and care they need. Here are examples of indicators of access to needed services that are of particular relevance to people with disabilities; many are important to anyone.

• Facilities, medical equipment and communication equipment accessible to those with physical challenges, including visual and hearing impairments;
• Availability of transportation to care sites;
• Availability of convenient times of service;
• Timeliness of access to routine and urgent care (e.g., waiting time for appointments; waiting time after arrival);
• Access to needed tests and treatments without extensive delays or hassles waiting for prior approvals;
• Access to needed prescription medications (including medications not included in the MCOs formulary, although they are on the State's Medicaid formulary);
• Access to identified/selected primary care providers;
• Access to specialists;
• Access to services of special importance to persons with disabilities (e.g., durable medical equipment and provision for the prompt repair of such equipment; rehabilitation services; physical, occupational and speech therapy; home health care (both skilled and unskilled); respite care); and
• Availability of translation and interpretation services for non-English speakers (including people with speech and hearing disabilities who use American Sign Language).

C. Supporting Member Involvement in Decision Making and System Improvement

Increasingly, Americans want to be more active in their interactions with the health care system. People with disabilities have been pioneers in pursuing a more active role and greater autonomy. In this domain, there are two levels at which it is important for MCO members with disabilities to be involved in decision making and system improvement. The first level is the individual patient's interaction with the MCO and their health care providers. The second level is the policy decisions that affect or guide a given MCO, or the entire program of care for people with disabilities.

Involvement in Decisions and Improvements at the Individual Level

• Provision of understandable information on how to get needed services and resolve member problems;
• Provision of understandable information on methods for filing complaints and grievances;
• Provision of understandable information on the definition of medical necessity used by the plan, on the criteria applied to determine medical necessity, and on the process used in making medical necessity decisions, including the process for appealing such decisions in a timely manner;
• Involvement of members in treatment planning for management of ongoing conditions;
• Provision of information on all treatment options; involvement of members in decisions regarding choice of treatments and specialty providers; and
• Support for the development and implementation of advanced directives.

Involvement in Decisions and Improvements at the System Level

• Inclusion of members, family and other caregivers and advocates on MCO governing and advisory boards;
• Inclusion of members, family and other caregivers and advocates on task forces and other groups organized by MCOs, State Medicaid agencies or others, to plan or track the use of managed care for people with disabilities;
• Participation of members, caregivers and advocates in the planning of quality improvement activities, including setting priorities on issues and conditions that should be the focus of such efforts; and
• Systematic use of methods to gather qualitative and quantitative information from members, caregivers and advocates to identify their priorities and their experiences using the system.

D. Resolving Member Problems and Concerns

Many, though not all, MCOs are relatively inexperienced in serving people with disabilities. For this reason if no other, it is critical to determine whether MCOs have put themselves in a position to learn quickly from their experiences and to correct their mistakes. One place where this would show is in how well they resolve the problems and concerns of particular members. In addition, as already noted, some health care needs of people with disabilities are rare or unexpected, and may only come to the surface in a crisis situation. Here are examples of indicators of whether an MCO is resolving member problems and concerns:

• Timely, courteous and helpful responses from plan customer service to member requests for information about the plan and how to use it;
• Timely, courteous and helpful responses from plan customer service to member requests to resolve questions and problems regarding access to care;
• Rapid response to resolve problems involving exceptional needs and circumstances;
• Use of appropriate procedures to hear and address member appeals, complaints and grievances (including rapid responses); and
• Reductions over time in the rate of disenrollment for cause by members with disabilities.

E. High Quality Interpersonal Interactions Between Members and Providers

Extensive consumer research indicates that everyone values highly the opportunity to have high quality interpersonal interactions with people in health care systems. There is also evidence that good patient-provider relationships have a positive effect on the management of long-term conditions and on the outcomes of health care. This domain of performance is an example of one that is important to everyone and especially important to and for people with disabilities. Here are examples of indicators in this domain:

• Proactive engagement of new members with primary care providers they have selected from the plan network;
• Providers who listen carefully to members and their caregivers;
• Providers who explain things clearly to members and their caregivers;
• Providers who treat members and their caregivers with courtesy and respect;
• Providers who spend enough time with members and their caregivers; and
• Medical office staff who are courteous, helpful and respectful of members and their caregivers.

F. Using Preventive Services to Keep Members Healthy and Functioning

This is another domain which is important to everyone and also very important for people with disabilities. Equity considerations are of special relevance here: it is critical that those with disabilities get the services we know are effective in prevention or early identification of various diseases, even though it may take special efforts to ensure they get these services at least as often as people with no disabilities. This remark applies to our first general indicator:

• Delivery of age- and gender-appropriate preventive health services.

  Considerable work has been done to identify, across the age spectrum, and by gender where relevant, specific primary and secondary prevention services and interventions of known value. Many measurement systems have identified sub-sets of these services of particular importance. However, for people with disabilities, there is another kind of prevention that is important, what public health people call tertiary prevention, i.e., the prevention of unnecessary complications and unnecessary deterioration in functioning for people who already have a defined condition. Our second general indicator is relevant to this kind of prevention. Unfortunately, less work has been done to specify and bring together services which are effective in this kind of prevention, although some have certainly been identified and are already in use as performance measures.

• Delivery of services to maintain and enhance functioning.

G. Coordinating and Integrating Medical and Non-Medical Services

Coordination of care has been consistently identified as critical for people with disabilities. One of the greatest hopes many have about managed care is that it will do a better job of coordinating and integrating care. The examples below provide different perspectives on coordination that can all be important to people with disabilities:

• Continuity of primary care and specialty providers over time;
• Ongoing coordination of medical services provided to a member by multiple providers;
• Coordination of physical and mental health/substance abuse service providers;
• Identification of non-medical services needed to maintain and enhance member health, functioning and autonomy; referral and follow-up to support delivery of such services;
• Referral to plan- and community-based psycho-social support and patient education services; and
• Ongoing coordination with providers of critical non-medical services that affect health and functioning.

H. Using State of the Art Treatments

This domain of performance comes closest to traditional quality assurance and quality improvement efforts. The domain emphasizes process measures, indicators that health care providers are using the right diagnostic and treatment procedures, medications, etc., in caring for their patients. To develop and select performance measures in this domain, it is essential to focus on a limited number of specific conditions that are sufficiently common to permit meaningful data collection and that have significant health consequences. These conditions can be identified in two different manners. The first is to build on the work already done by multi-stakeholder groups working with experts (see for example the Sentinel Conditions for Medicaid MCOs developed collaboratively by plans, State agencies and HCFA). Alternatively, the set of conditions could also be identified through multi-stakeholder interactions in a particular State. For these conditions, the strategy would then be to identify indicators that MCOs:

• Use interventions known to be of importance in the management and resolution of significant medical care problems; and
• Use interventions known to be of importance in the management and resolution of significant behavioral health problems

I. Improving the Outcomes of Care

Many believe that the most significant performance measures are those that reflect the outcomes of care, that is, whether people get better and how quickly; whether their conditions are managed effectively even if they cannot be cured; whether their functioning and quality of life are maintained to the extent possible; and whether and when they die. In addition to these traditional definitions of good outcomes, which tend to be driven by the expectations of professionals, many people believe there is another important outcome of care: the degree to which patients or plan members are satisfied with the care they receive.

In spite of the acknowledged importance of medical outcomes, project staff were consistently advised, even by those who have devoted their lives to the development of outcome measures and the conduct of outcomes research, that it would not be sensible to include medical outcome measures in our list of performance measures. Outcome measures have been used primarily in highly controlled research studies, in which an identified group of individuals are carefully tracked over time to see how they respond to a new treatment, as compared to either no treatment or a more conventional treatment. Care is taken to ensure that the groups of people who get the different kinds of treatment are extremely similar to one another. This is all to ensure that any differences in outcomes can, with confidence, be attributed to differences in the treatment.

However, these controlled circumstances do not exist when MCO performance is being measured. First, there is no guarantee that the people with disabilities who enroll in one MCO are going to be at all similar to the people with disabilities who enroll in another, or who remain in Medicaid FFS or PCCM systems. Second, there is no guarantee that these people will stay in an MCO so their progress (or lack of progress) can be tracked. Finally it is important to choose performance measures that address things the MCO can strongly influence. Many factors besides medical care are known to have a strong influence on outcomes. Research studies are carefully structured to track these other influences and take them into account, but it is very difficult and expensive to set up these controls in examining a population of MCO member.

Given these considerations, this domain should emphasize that MCOs are:

• Demonstrating improvements over time in the overall satisfaction, among members who have disabilities, with the plan, its providers, and the quality of care they are receiving.

In addition, MCOs can and should conduct their own outcome studies. Another indicator in this Domain would therefore be the following:

• With respect to conditions of special importance to persons with disabilities, the MCO conducts studies to:

  • Support or demonstrate improvements over time in the health outcomes of care;
  • Support or demonstrate improvements over time in the functional status of members; or
  • Support or demonstrate improvements over time in the quality of life of members.

IV. SPECIFIC MEASURES TO ASSESS THE PERFORMANCE OF MANAGED CARE IN SERVING PEOPLE WITH DISABILITIES

This Chapter includes measures and measurement systems that should be considered for use in assessing the performance of MCOs in serving people with disabilities. The Chapter begins by describing how measures were selected. The Chapter then presents a matrix that allows the reader to identify which major domains of measurement are addressed by each measure or measurement system. For example, if you are interested in learning about HEDIS® measures, the matrix will show which major domains of performance are, and are not, covered by these measures. After the matrix, we present, for each major domain, the specific items in the domain that are available in different measures and measurement systems. For example, if you are interested in the domain of Providing Access to Needed Services, you should consult this section to get a quick overview of the specific access measures available. Based on these summaries, we then comment briefly on which domains have extensive measures and which do not, with recommendations for measurement development.

The Chapter then moves to details about each measure. A standard set of information is presented for each measure, and these items of information are described to orient the reader. Finally, each measure or measurement system is presented.

A. How We Identified and Selected the Measures in the Guide

The measures in the Resource Guide were identified through an extensive search process that included the following:

• a review of published literature;
• a review of unpublished literature, including the proceedings and hand-out materials for relevant conferences; and
• in-depth interviews with experts in performance measurement, and in the delivery of health care services to people with disabilities.

In selecting the measures to be included in the Resource Guide, we used the following criteria:

• High relevance to the domains of performance included in our framework (see Chapter Three);
• Appropriateness as a measure of the performance of a system of care rather than for the evaluation of individual health care providers;
• Applicability to a population of people with disabilities with a broad mix of illnesses and impairments;
• Sufficient specificity for consistent measurement to occur over time and across settings;
• Evidence of scientific reliability and validity, as demonstrated either through the procedures used to develop the measure or the results of psychometric tests conducted after initial use of the measure;
• History of development or use of the measure with people with disabilities;
• Ease, cost and burden of data collection;
• Availability of the measure in the public domain;
• Availability of technical documentation for the measure and information on how data for the measure should be collected; and
• Availability of technical support for the use of the measure, preferably at low or no cost.

All the measures do not meet all these criteria. Criteria in boldface are met by all measures in the Guide. With respect to other criteria, some measures rate more highly than others, the Guide includes information that permits readers to make their own assessments on these criteria. A few measures are not available as we go to press, but are expected to be available very shortly.

B. Summary Matrix of Domains Addressed in Each Measure

Table 1 below matches each of the five measurement systems to the nine quality domains we have created. The following is a listing of the measurement systems and their abbreviations as referenced in the table:

• The Health Plan Employer Data and Information Set (version 3.0) of the National Committeee on Quality Assurance: HEDIS®
• Performance Measures for Managed Behavioral Healthcare Programs (version 2.0) of the American Managed Behavioral Health Association: PERMS
• The Consumer Assessment of Health Plans Survey: CAHPS
• The Oregon Health Plan Adult Satisfaction Survey of 1997 of the Oregon Human Resources Department: Oregon
• The Picker Institute Surveys of Adult Medical/Surgical Hospital Stays; Rehabilition Programs; and Home Care Services of the Picker Institute: Picker

TABLE 1: Summary Matrix of Domains Addressed in Each Measure
Domains	Measures
	HEDIS®	PERMS	CAHPS	Oregon	Picker
1. Creating a System with the Right Capacities	X	-	X	X	-
2. Providing Access to Needed Services	X	X	X	X	X
3. Supporting Member Involvement in Decision Making and System Improvement	-	-	X	-	X
4. Resolving Problems and Concerns	X	-	X	X	X
5. High Quality Interpersonal Interactions Between Members and Providers	-	-	X	X	X
6. Using Preventive Services to Keep Members Healthy and Functioning	X	-	X	X	-
7. Coordinating and Integrating Medical and Non-Medical Services	X	-	X	X	X
8. State of the Art Treatments	X	X	-	X	X
9. Improving the Outcomes of Care	X	X	X	X	X

C. Summary of Specific Items Available in the Measures for Each Domain

In this section, we list, for each domain in our framework, the specific items that are measured in each measure or measurement system included in the Resource Guide. Detailed descriptions of these measures and measurement systems can be found in Section F below.

Domain 1. Creating a System with the Right Capacities

HEDIS®

• Pediatric mental health network
• Chemical dependency services
• Quality assessment and improvement systems
• Provider turnover

CAHPS

• Ability to find a personal doctor or nurse

Oregon Health Plan Survey

• Ability to have a choice among different plans
• Ease of communication between personal doctor and specialists
• Creating a position of Exceptional Needs Care Coordinator (ENCC) in health plan
• Rating of helpfulness of ENCC and ENCC services

Domain 2. Providing Access to Needed Services

HEDIS®

• Mental health utilization--percentage of members receiving inpatient day/night and ambulatory services
• Mental health utilization--inpatient discharges and average length of stay
• Chemical dependency utilization--percentage of members receiving inpatient, day/night and ambulatory services

PERMS

• Telecommunications standards--standards such as a call abandonment rate of less than 5% of calls on-hold for less than 30 seconds, and an average answer time speed of five rings or fewer
• Consumer satisfaction with the time interval to the first appointment

CAHPS

• Problems getting care you and your doctor thought was needed
• Problems with delays in treatment while waiting approval from the health plan
• Access to needed help and advice from the doctor's office, by telephone
• Timely access to appointments for regular or routine care
• Timely access to a doctor or health care provider for urgently needed care
• Access to specialists
• Access to interpreters (Medicaid surveys)
• Access to special medical equipment (surveys for Medicare and people with disabilities)
• Access to physical, occupational and speech therapy (Medicare HMO survey and surveys for people with disabilities)
• Access to home health care services (Medicare HMO survey and surveys for Medicare and people with disabilities)
• Access to respite care (Medicare HMO survey and surveys for people with disabilities)
• Access to prescription medications (Medicare surveys)

Oregon Health Plan Survey

• Ease of access to a particular primary care provider
• Ease of access to specialist doctors
• Means of transportation to primary health care place
• Ease of travel to primary health care location
• Use of interpreter if needed (foreign languages, sign languages)
• Medications packaged for ease of use
• Convenient locations of pharmacies
• Ease or difficulty of early appointments, conveniently timed appointments, appointments with specialists, prescription medicines, medical advice
• Ease or difficulty of obtaining emergency medical care
• Ease or difficulty of maneuvering around medical offices (if blind, visually impaired, or use special equipment to move from place to place)
• Ease of access to counseling/mental health services
• Ease of access to alcohol or drug treatment services
• Ease of obtaining physical, occupational or speech therapy
• Ease of obtaining home health care
• Ease of acquisition of special medical equipment
• Ease of repair of special medical equipment

Picker Institute Surveys

• Timely access to hospital admission from emergency room (Hospital Survey)
• Availability of health care providers when and as frequently as needed
• Timely performance of needed tests and procedures (Hospital and Rehabilitation Surveys)
• Convenience of timing of home care visits (Home Care Survey)
• Timely availability of pain medication (Hospital Survey)
• Availability of needed special equipment in the home, and changes to home (Rehabilitation and Home Care Surveys)
• Access to sufficient number of home care visits (Home Care Survey)

Domain 3. Supporting Member Involvement in Decision Making and System Improvement

CAHPS

• Involvement in decision making about care (child surveys address involvement of both child and parent; Behavioral Health Survey may address involvement of family in decision making)
• Accuracy of information provided prior to enrollment (Medicaid surveys)
• Access to written information from plan about covered services (Medicaid surveys)

Picker Institute Surveys

• Patients have enough to say about their treatment

Domain 4. Resolving Problems and Concerns

HEDIS®

• Disenrollment rates

CAHPS

• Ability to get needed information from health plan customer service
• Resolution of complaints and appeals (Medicaid, Medicare surveys)
• Reasons for disenrollment (Disenrollee survey)

Oregon Health Plan Survey

• Notification by health plan and/or medical assistance worker of rights to complain or appeal
• Access to information from membership services
• Access to easily understandable information about benefits and services
• Knowledge about Ombudsman staff at the Office of Medical Assistance Programs (OMAP)
• Ease of access to OMAP's Ombudsman staff
• Helpfulness of information received from Ombudsman staff

Picker Institute Surveys

• Timely response to complaints (Home Care Survey)
• Agency willingness to change home care providers (Home Care Survey)

Domain 5. High Quality Interpersonal Interactions Between Members and Providers

CAHPS

• Providers listen carefully (in children's surveys, questions are asked re parent and child)
• Providers explain things clearly (in children's surveys, questions are asked regarding parent and child)
• Providers spend enough time
• Providers show respect for what patients say (in children's surveys, questions are asked regarding parent and child)
• Medical Office Staff are respectful and courteous
• Medical Office Staff are helpful
• Problems interacting with providers because of language difficulties (Medicaid surveys)
• Providers offer reassurance and support to parents regarding how they care for their child (Children with Special Needs survey)

Oregon Health Plan Survey

• Providers listen without interrupting or rushing
• Providers explain things clearly
• Providers show respect for patient
• Providers spend sufficient time with patient
• Providers follow through on test results or after care
• Health care professionals do not convey conflicting information
• Providers provide enough information about a health care condition
• Ratings of personal doctor or nurse

Picker Institute Surveys

• Providers listen carefully
• Providers explain treatments and answer questions clearly
• Staff and providers are courteous and helpful
• Home care providers are courteous to family and friends (Home Care Survey)
• Staff explain reasons for delays (Hospital Survey)
• Providers treat patients with respect and dignity
• Patients have enough to say about their treatment
• Providers explain risks and benefits of surgery and how patient will feel after surgery (Hospital Survey)
• Providers discuss patient's anxieties and fears and/or offers encouragement
• Providers inspire confidence and trust
• Providers explain test results (Hospital Survey)
• Providers talk to, involve, provide information to family members
• Providers do not talk about patient in their presence as if they were not there
• Patients get enough privacy (Rehabilitation Survey)

Domain 6. Using Preventive Services to Keep Members Healthy and Functioning

HEDIS®

• Advising smokers to quit
• Flu shots for older adults
• Cervical cancer screening
• Breast cancer screening
• Childhood immunization status
• Adolescent immunization status

CAHPS

• Advising smokers to quit (Medicare, Medicaid surveys)
• Flu shots for older adults (Medicare survey)
• Provider discussion of child development (Children with Special Needs survey)
• Provider encouragement of preventive health behaviors (Children with Special Needs survey)
• Reminders about check-ups and preventive care (Medicaid Children's surveys)

Oregon Health Plan Survey

• Provider discussion of prevention and maintenance
• Provider reminder of mammogram, pap test, quitting tobacco, cholesterol test, weight control, prostate screening, alcohol or drug screening

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

HEDIS®

• Arrangements with public health, educational and social service organizations
• Case management

CAHPS

• Availability of case management (Children with Special Needs survey)
• Linkage to schools (Children with Special Needs survey)

Oregon Health Plan Survey

• Arrangements to "watch kids" at home or provide transportation
• Patient gets help with coordination of medical care needs (from medical provider, from Exceptional Needs Care Coordinator (ENCC) at health plan, from Case Manager at social services, other)
• Rating of helpfulness of provider in giving information about other agencies or services

Picker Institute Surveys

• Providers are consistent in what they say to patients (Hospital Survey)
• Presence of one person to coordinate care provided by rehabilitation team (Rehabilitation Survey)
• Support for family involvement in providing care, including care post-discharge (Rehabilitation Survey)
• Arrangements made for post-discharge care from other agencies (Rehabilitation Survey)
• Consistency of provider across time (Home Care Survey)
• Rating of instruction given to patient to take care of medical needs at home
• Patient ratings of provider coordination (Hospital and Rehabilitation Survey)

Domain 8. State of the Art Treatments

HEDIS®

• Beta blocker treatment after a heart attack
• Eye exams for people with diabetes
• Treating children's ear infections
• Prenatal care in the first trimester
• Check-ups after delivery
• Follow-up after hospitalization for mental illness

PERMS

• Encouragement by provider to use self-help or consumer-run programs
• Medication management for individuals with schizophrenia
• Family visits for children undergoing mental health treatments
• Ambulatory follow-up within 7 and 30 days of discharge for mental health
• Ambulatory follow-up within 7 Days of discharge for substance abuse
• Ambulatory follow-up after hospitalization for major depressive disorder

Oregon Health Plan Survey

• Rating of home health care

Picker Institute Surveys

• Hospitals provide machinery patients can use to give themselves pain medication (Hospital Survey)
• Staff does all they can to help patients control pain (Hospital and Home Care Surveys)
• Patients are given information on discharge about medications and how to take them, danger signals and other follow-up care (Hospital and Rehabilitation Surveys)
• Patients are given information on nutritional needs, pain control, etc. (Hospital and Rehabilitation Surveys)

Domain 9. Improving the Outcomes of Care

HEDIS®

• Readmission for specified mental health disorders
• Readmission for chemical dependency

PERMS

• Rate of engagement with treatment for substance abuse

CAHPS

• Ratings of personal doctor
• Ratings of specialists
• Ratings of all health care providers
• Ratings of health plan

Oregon Health Plan Survey

• Patient rating of health status compared to one year previously

Picker Institute Surveys

• Patient ratings of hospital care (Hospital Survey)
• Patient ratings of rehabilitation services (Rehabilitation Survey)
• Patient ratings of home care (Home Care Survey)

D. Areas Where Measures Are and Are Not Available

The preceding summary of the specific measures available in each domain reveals clearly that in certain domains there is a rich and varied choice of measures, while in others there are relatively few measures. There is an especially wide choice of measures in these areas:

• Domain 2. Providing Access to Needed Services; and
• Domain 5. High Quality Interpersonal Interactions Between Members and Providers.

Research has indicated that these two areas are of special importance to all kinds of patients and consumers, in the context of managed care, and they are certainly of importance to people with disabilities. In addition, it is easy to gather data on these topics from patients and consumers themselves, who are often in the best position to provide useful information on these topics.

There is a reasonable number of measures available in these two areas:

• Domain 6. Using Preventive Services to Keep Members Healthy and Functioning; and
• Domain 8. State of the Art Treatments.

These two areas have been a focus of a good deal of measurement development efforts that build on the foundations of measurement of clinical quality of care. However, there is still work needed in these areas to develop measures that address conditions and concerns specific to people with disabilities, including more and better measures of mental health and developmental problems. In contrast, there are relatively few measures available in Domain 9, Improving the Outcomes of Care. As noted earlier, there are methodological difficulties in applying traditional clinical outcome measures in the assessment of the outcomes of care for a population of members in MCOs. In the early stages of the process of measuring the performance of MCOs in caring for people with disabilities, other arenas may therefore have higher priority for new measurement development.

In particular, the scarcity of measures in the following areas is of concern:

• Domain 1. Creating a System with the Right Capacities
• Domain 3. Supporting Member Involvement in Decision Making and System Improvement
• Domain 4. Resolving Problems and Concerns; and
• Domain 7. Coordinating and Integrating Medical and Non-Medical Services.

Several of these domains can and often are addressed by looking at MCO structures. Chapter Five includes several criteria sets that specify structural and procedural characteristics of MCOs that are believed to be necessary, if not sufficient, for the delivery of high quality care to people with disabilities. As we note, criteria are not, in and of themselves, measures, but the judicious use of structural criteria can help to complement other measures.

However, some aspects of these domains cannot be addressed simply in structural terms. First, we need to know if structures are not only in place, but working to meet the needs of people with disabilities. Second, some aspects of these domains go beyond structural issues. Project staff believe that it is especially critical that priority be given to measurement development in the following areas:

• the involvement of individual people with disabilities (and where appropriate their family and other caregivers) in decisions about their health care;
• the involvement of people with disabilities, their families and other caregivers, and their advocates, in decisions about the design and improvement of health care delivery systems;
• the degree to which member problems, concerns and grievances are promptly addressed and resolved to their satisfaction;
• continuity and coordination of medical care services; and
• coordination of medical and related social and ancillary services.

E. Overview of Information to be Provided About Each Measure or Measurement System

The following information is presented below, wherever it is available, for each measure and measurement system in the Resource Guide:

Name of Measure: The formal name of the measure or measurement system

The Author or Organization That Developed the Measure: This would include the key contact person and location information for where to get a complete copy of the measure and any documentation that is available for the measure.

Domains of Performance Addressed by the Measure: Many of the measures in the Resource Guide address more than one of the domains of performance we have discussed in Chapter Two, Section E. We will list each of the domains of performance to which the measure is relevant and usually give examples of specific items or indicators in each domain that are addressed in the measure.

Data Collection Strategies Required for Use of the Measure: This includes the following issues:

• What type of data collection is required?
• From whom are data collected?
• What kind of sampling or oversampling is needed?
• What if any methods are available for risk-adjustment of the data collected?

What Evidence is Available to Support the Validity and Reliability of Measure: The Guide summarizes evidence drawn from the process of developing the measure and/or from psychometric testing of the measure that indicates that it is likely to be valid and reliable.

What Are the Populations and Settings in Which the Measure Has Been or Could Be Used: Very few performance measures have been developed specifically to assess care for people with disabilities. This section provides information on the populations and settings for which the measure was originally developed, and where it has been used, and gives an assessment of the additional populations or settings where it could be used.

What Written Documentation is Available to Support Use of Measure: This section lets the reader know whether there is written documentation that can be used in implementing the measure, and the level of detail of this documentation.

What Consultation is Available to Support Use of Measure: This section indicates whether technical consultation is available, from the original developers of the measure or from others, about its use. When consultation is only available at a charge this is noted.

Limits on Use of the Measure: This final section presents any limitations on the use of the measure, or on the interpretation of results, that need to be taken into consideration in a performance measurement context.

F. Descriptions of Measures and Measuring Systems

1. Selected Measures from the Health Plan Employer Data and Information Set (HEDIS®), Version 3.0

HEDIS® is one of the most well-known performance measurement systems for MCOs. It is a measurement system that includes dozens of specific measures. We have selected, from HEDIS® 3.0, those measures that are most relevant to assessing the performance of MCOs in caring for persons with disabilities.

Name of Measurement System: HEDIS®3.0

Author/Developer: National Committee for Quality Assurance (NCQA), 2000 L Street, N.W., Suite 500, Washington, D.C. 20036, Phone (202) 955-3500

NCQA is a non-profit organization formed specifically for the purpose of assessing and improving the quality and performance of health maintenance organizations. It is governed by a Board of Directors that includes representatives from purchasers of health care, from MCOs, and from the public. NCQA began its work by developing standards for the independent accreditation of HMOs. It then moved into performance measurement, with the HEDIS® system. HEDIS® 3.0 includes measures considered relevant for the commercially insured population, for people on Medicaid, and for people on Medicare. It was developed by the NCQA Committee on Performance Measurement (CPM), whose membership mirrors that of the NCQA Board.

Domains of Performance Addressed by the Measure:

HEDIS® 3.0 measures are organized by NCQA into sub-sets which do not map with our domains. We have chosen specific measures from several sub-sets, including the following: (1) effectiveness of care; (2) health plan descriptive information; (3) health plan stability; and (4) use of services. Using our domains of performance, we will list the specific HEDIS® 3.0 measures we recommend:7

Domain 1. Creating a System with the Right Capacities

• Pediatric mental health network
• Chemical dependency services
• Quality assessment and improvement systems
• Provider turnover

Domain 2. Providing Access to Needed Services

• Mental health utilization--percentage of members receiving inpatient day/night and ambulatory services
• Mental health utilization--inpatient discharges and average length of stay
• Chemical dependency utilization--percentage of members receiving inpatient, day/night and ambulatory services

(Note: Utilization rates can only be interpreted if there is a well-grounded standard for determining what the "correct" level of utilization should be for a given population of individuals. For example, if utilization rates are compared over time or across plans, there needs to be confidence that the rate at which members in different plans need a specific service is very similar. Even in that cases, only wide disparities would generate cause for concern.)

Domain 4. Resolving Problems and Concerns

• Disenrollment rates

Domain 6. Using Preventive Services to Keep Members Healthy and Functioning

• Advising smokers to quit
• Flu shots for older adults
• Cervical cancer screening
• Breast cancer screening
• Childhood immunization status
• Adolescent immunization status

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

• Arrangements with public health, educational and social service organizations
• Case management

Domain 8. State of the Art Treatments

• Beta blocker treatment after a heart attack
• Eye exams for people with diabetes
• Treating children's ear infections
• Prenatal care in the first trimester
• Check-ups after delivery
• Follow-up after hospitalization for mental illness

Domain 9. Improving the Outcomes of Care

• Readmission for specified mental health disorders
• Readmission for chemical dependency

Data Collection Strategies Required for Use of the Measure:

Type of Data Collection

The measures listed above require a wide variety of data collection strategies, but are primarily based on administrative and clinical records.

From Whom Are Data Collected

All data are collected by the MCO, using the detailed specification provided by NCQA. It is essential to validate these data using an independent auditor. NCQA is currently training and licensing organizations to serve this role.

Sampling and Oversampling Issues

Hundreds of MCOs collect HEDIS® data, either on their entire population or on members supported by particular sponsors (private employers, Medicare, etc.). Especially since NCQA earlier developed a version of HEDIS® specifically for Medicaid (which has now been integrated into HEDIS® 3.0), many MCOs collect HEDIS® data on Medicaid populations. However, it is very likely that if a routine general population sample is taken of MCO members for purposes of HEDIS®, the number of people with disabilities in the sample will not be sufficient to permit meaningful conclusions. It is therefore essential either to target data collection at members who are known (for example by their eligibility category) to have a disability, or to develop methods for "oversampling" the records of people with disabilities.

Risk Adjustment

The documentation for HEDIS® 3.0 includes information on whether and how specific measures can be risk adjusted.

Evidence to Support Validity and Reliability of Measures:

HEDIS® measures are selected using a rigorous and extended process of examination and testing of candidate items against criteria of scientific validity and reliability. The documentation for each measure includes detailed information on the specific evidence of validity and reliability.

Populations and Settings in Which Measures Have Been/Could Be Used:

The original HEDIS® measures were designed to assess care provided to commercially insured populations enrolled in HMOs. HEDIS® 3.0 incorporates measures targeted to Medicaid and Medicare populations. As a measurement system, HEDIS® 3.0 is intended to address a cross section of the American people. To the extent that people with disabilities have been enrolled in various kinds of MCOs (commercial, Medicaid, Medicare) the measures have been used with this group. Given appropriate sampling and oversampling, HEDIS® 3.0 measures can be used specifically to assess care for people with disabilities.

The extensive use of HEDIS® among populations without disabilities facilitates comparisons of MCO performance for people with disabilities to their performance for other populations. For example, HCFA has mandated the collection of HEDIS® 3.0 data by all Medicare risk-contract HMOs serving Medicare beneficiaries (including those eligible because of a disability).

NCQA has developed "Quality Compass," a system through which HMOs voluntarily provide the results of HEDIS® measurement to the agency, for incorporation into a data base that provides information on performance on individual measures for a substantial sample of HMOs. This can be useful in comparing performance and in developing benchmarks. However, because HEDIS® measures are specific to HMOs, they do not permit comparisons with FFS or PCCM.

Written Documentation Available to Support Use of Measure:

Extensive and detailed documentation is available from NCQA on individual HEDIS® 3.0 measures and on the entire HEDIS® data collection process. Since the documentation is voluminous, there is a charge for it.

Consultation Available to Support Use of Measures:

Some consultation is available from NCQA regarding the use of HEDIS® 3.0. As noted above, NCQA is currently identifying organizations qualified to validate HEDIS® data collection efforts. In addition, NCQA has begun to work on the dissemination of HEDIS® information, and in the design of materials to report the results of HEDIS® to different audiences, including consumers. Perhaps more significant, hundreds of HMOs have used the system and are familiar with it, and many private and public purchasers, including State Medicaid agencies, have also used the system and could provide information regarding their experiences on an informal basis.

Limits on Use of the Measures:

Limits on the use of utilization rate measures have already been noted. It should also be noted that descriptive information provided by HMOs, such as that included in Domains 1 and 7, is quite difficult to validate. The same is true for all forms of self-reported structural data.

2. Selected Measures from the Performance Measures for Managed Behavioral Healthcare Programs (PERMS)

Name of Measurement System: PERMS 2.0 (Performance Measures for Managed Behavioral Healthcare Programs)

Author/Developer: American Managed Behavioral Healthcare Association (AMBHA), 700 Thirteenth Street, NW, Suite 950, Washington, D.C. 20005, Phone (202)434-4565, Contact: E. Clark Ross, D.P.A., Executive Director

Founded in 1994, AMBHA is an association of managed behavioral health organizations (MBHOs) based in Washington, D.C. The member organizations, from 38 States, manage the mental health and substance abuse health plan benefits for over 98 million people. The association was formed to work on issues of public accountability, quality, and public policy and communication. The mission statement of the organization has as one of its objectives: "To promote health plan, management agent, and provider accountability through performance measurement."

At the time PERMS 1.0 was first administered, AMBHA included 19 member organizations, of which 13 responded to parts of PERMS; these firms had covered lives of approximately 21 million people. As PERMS 2.0 evolved, portions of it were incorporated within the HEDIS® measures released by the NCQA in January of 1997. Developmental work on PERMS 2.0 is still ongoing, and reflects areas "requiring more attention" such as substance abuse, the severely and persistently mentally ill, and special populations. As AMBHA develops and refines measures, their goal is to "act as a testing site and a prod" so that quality measurement organizations that operate on a national level will incorporate what AMBHA considers to be valid and realistic behavioral health performance measures. The NCQA has incorporated a number of these measures into their HEDIS® Measurement system; other measures for consumer satisfaction may be incorporated into the Consumer Assessment of Health Plans (CAHPS) surveys, in particular the Behavioral Health Survey, which are discussed in section 3 below.

Domains of Performance Addressed by the Measure:

PERMS 2.0 categorizes its measures into Access to Care, Quality of Care, and Consumer Satisfaction under the headings of "Core Measures," and "Leadership Measures." The Core Measures domain consists of "HEDIS-type" measures, with the addition of an item on "call abandonment reporting." The Leadership Measures domain incorporate HEDIS-type measures for diagnoses of depression, schizophrenia, and alcohol abuse, by age and by gender.

Domain 2. Providing Access to Needed Services

• Telecommunications standards--standards such as a call abandonment rate of less than 5% of calls on-hold for less than 30 seconds, and an average answer time speed of five rings or fewer
• Consumer satisfaction with the time interval to the first appointment

Domain 8. State of the Art Treatments

• Encouragement by provider to use self-help or consumer-run programs
• Medication management for individuals with schizophrenia
• Family visits for children undergoing mental health treatments
• Ambulatory follow-up within 7 and 30 days of discharge for mental health
• Ambulatory follow-up within 7 days of discharge for substance abuse
• Ambulatory follow-up after hospitalization for major depressive disorder

Domain 9. Improving the Outcomes of Care

• Rates of successful engagement in treatment for substance abuse

Data Collection Strategies Required for Use of the Measure:

Type of Data Collection

PERMS data are collected through mailed surveys of which are sent to Harvard Medical School for analysis.

From Whom Are Data Collected

Data are collected from member organizations of AMBHA who respond based on their administrative and clinical records. These members in turn collect data from their patients about satisfaction. AMBHA has over 48,000,000 enrollees across 38 States.

Sampling and Oversampling Issues

The AMBHA Managed Behavioral Healthcare Quality and Access Survey was sent to all members; the patient population of these organizations accesses behavioral health services through from multiple sources: 26 percent are covered by employee assistance programs; 27 percent by managed indemnity/utilization review models; 27 percent by HMOs, and 20 percent by network-based models. There is no breakdown of Medicaid or commercial subpopulations. Special arrangements would be needed to identify and target or oversample people with disabilities in the patient populations of AMBHA members in order to get a sufficient sample to provide information specifically on this group.

Evidence to Support Validity and Reliability of Measures:

The original measures used in PERMS were developed by the AMBHA Policy Subcommittee on Quality and Access Standards, which has seven working groups. The groups used research and their own knowledge to develop the indicators to measure quality and access. A contracted survey firm, Foster Higgins, reviews the questions for wording and structure "in accordance with accepted research standards." The measures which are part of other measurement sets (such as HEDIS®) are tested as part of their protocol. "The large sample size assures the statistical validity of the information and the ability to generalize information to the rest of the population," according to the Foster Higgins firm which conducted the survey.

Populations and Settings in Which Measures Have Been/Could Be Used:

Measures are considered by AMBHA to be appropriate for all patients of MBHOs including Medicaid populations.

Written Documentation Available to Support Use of Measure:

Sections encapsulated by HEDIS® will have the support of NCQA. A list of AMBHA reports, studies, and publications and information on how to obtain these documents is available at their web page: http://www.ambha.org.

Consultation Available to Support Use of Measures:

The availability of consultation for use of PERMS outside of AMBHA's membership is unclear, but presumably AMBHA staff would be available to their membership for assistance.

Limits on Use of the Measure:

The results of the "report card" from PERMS 1.0 is available to the public and to health care purchasers. Results on PERMS 2.0 will also be available to the public upon publication.

Underlying all measurement of behavioral managed care performance is a "fundamental lack of consensus about what constitutes quality of care in the behavioral arena" (Gore, 1998); therefore, the use of the measure may be limited by differing opinions of the definition of good care in the behavioral health field. In addition, consultation would be needed with AMBHA and the experts with whom it works to develop an approach to the use of PERMS measures by sponsors other than behavioral health care organizations.

3. Consumer Assessment of Health Plans (CAHPS) 2.0 Surveys

Name of Measurement System: Consumer Assessment of Health Plans (CAHPS) Surveys

Author/Developer: The CAHPS surveys, which are designed to gather information from people who have been in and used health plans about their experiences in the plans, and their ratings of the plans, are being developed by three teams of researchers. Here are the three organizations leading the teams, with the name of the Principal Investigator(s).

Paul Cleary, Ph.D., Department of Health Care Policy, Harvard Medical School, 180 Longwood Avenue, Boston, MA 02115-5899

Ronald Hays, Ph.D. and Pamela Farley Short, Ph.D., The RAND Corporation, 1700 Main Street, Santa Monica, CA 90407-2138

James Lubalin, Ph.D., Research Triangle Institute (RTI), 1615 M Street, N.W., Suite 740, Washington, D.C. 20036-3209

The CAHPS project has been conducted with initial and primary support from the Agency for Health Care Policy and Research (AHCPR), with supplemental support from HCFA and private foundations. In addition to their support of the three CAHPS teams listed above, AHCPR has also created and is supporting a Survey User's Network (SUN network), whose role is to disseminate the CAHPS surveys and the reporting templates associated with these surveys, and to provide hands on support to those interested in sponsoring CAHPS surveying efforts. The organization operating the SUN network is Westat, Inc., 1650 Research Blvd., Rockville, MD 20850, Primary Contact Person: Vasudha Narayanan. The CAHPS Hotline is (800) 492-9261

AHCPR's website has information on CAHPS. By accessing http://www.ahcpr.gov/qual you can get both basic and detailed information on the project.

Domains of Performance Addressed by the Measure:

CAHPS is a measurement system, rather than a measure. It is a collection of surveys that focus on different groups of health plan members. Most but not all CAHPS surveys are designed for use with people enrolled in a wide range of health insurance plans, including MCOs of various kinds, and FFS plans. CAHPS surveys have been fully developed that address the following populations:

• Adults who are privately insured (this is considered the "core" CAHPS survey);
• Children who are privately insured;
• Adults who are eligible for Medicaid (with special emphasis on those eligible under TANF, i.e., women with children);
• Children who are eligible for Medicaid (again with special emphasis on the TANF population);
• Persons with chronic conditions and disabilities; and
• Medicare beneficiaries enrolled in HMOs.

Surveys are currently under development that address several other special populations or issues.

• Behavioral Health Services;
• Children with Special Needs;
• Medicaid SSI;
• Disenrollment from Health Plans by Medicare beneficiaries; and
• Medicare beneficiaries enrolled in traditional fee-for-service Medicare.

These surveys are not final, and certain items that appear in currently available drafts may not be present in the final version, or may be reworded.

CAHPS surveys address a number of domains of performance, highlighting those for which people in plans can be considered good sources of data. Under each domain covered by one or more CAHPS surveys, the kinds of questions relevant to that domain are listed. When a question is found only in one survey that is indicated.

Domain 1. Creating a System with the Right Capacities

• Ability to find a personal doctor or nurse

Domain 2. Providing Access to Needed Services

• Problems getting care you and your doctor thought was needed
• Problems with delays in treatment while waiting approval from the health plan
• Access to needed help and advice from the doctor's office, by telephone
• Timely access to appointments for regular or routine care
• Timely access to a doctor or health care provider for urgently needed care
• Access to specialists
• Access to interpreters (Medicaid surveys)
• Access to special medical equipment (surveys for Medicare and people with disabilities)
• Access to physical, occupational and speech therapy (Medicare HMO survey and surveys for people with disabilities)
• Access to home health care services (Medicare HMO survey and surveys for Medicare and people with disabilities)
• Access to respite care (Medicare HMO survey and surveys for people with disabilities)
• Access to prescription medications (Medicare surveys)

Domain 3. Supporting Member Involvement in Decision Making and System Improvement

• Involvement in decision making about care (child surveys address involvement of both child and parent; Behavioral Health Survey may address involvement of family in decision making)
• Accuracy of information provided prior to enrollment (Medicaid surveys)
• Access to written information from plan about covered services (Medicaid surveys)

Domain 4. Resolving Problems and Concerns

• Ability to get needed information from health plan customer service
• Resolution of complaints and appeals (Medicaid, Medicare surveys)
• Reasons for disenrollment (Disenrollee survey)

Domain 5. High Quality Interpersonal Interactions Between Members and Providers

• Providers listen carefully (in children's surveys, questions are asked for both the parent and child)
• Providers explain things clearly (in children's surveys, questions are asked for both the parent and child)
• Providers spend enough time with the patient
• Providers show respect for what patients say (in children's surveys, questions are asked for both parent and child)
• Medical Office Staff are respectful and courteous
• Medical Office Staff are helpful
• Problems interacting with providers because of language difficulties (Medicaid surveys)
• Providers offer reassurance and support to parents regarding how they care for their child (Children with Special Needs survey)

Domain 6. Using Preventive Services to Keep Members Healthy and Functioning

• Advising smokers to quit (Medicare, Medicaid surveys)
• Flu shots for older adults (Medicare survey)
• Provider discussion of child development (Children with Special Needs survey)
• Provider encouragement of preventive health behaviors (Children with Special Needs survey)
• Reminders about check-ups and preventive care (Medicaid Children's surveys)

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

• Availability of case management (Children with Special Needs survey)
• Linkage to schools (Children with Special Needs survey)

Domain 9. Improving the Outcomes of Care

• Member ratings of personal doctor
• Member ratings of specialists
• Member ratings of all health care providers
• Member ratings of health plan

Data Collection Strategies Required for Use of the Measure:

Type of Data Collection

The CAHPS 2.0 measurement system is a set of surveys. Each can be administered by either telephone or mail or a mix of the two methods. Testing has been conducted that indicates no differences in results between phone and mail administration. In order to administer a CAHPS survey it is essential to have a sampling frame of members of each health plan whose performance is being assessed. If a specialized survey is used, the sampling frame must be specific to the particular group of members for whom that survey is relevant. Note, however, that initial screening questions confirm the individual's membership in the plan.

From Whom Are Data Collected

CAHPS requires the collection of survey information from members of health plans. Typically, several health plans are surveyed because they are being offered either to a population (the privately insured; Medicaid eligibles) by a sponsor (a large employer or purchasing coalition; a State Medicaid agency) that decides to implement CAHPS. Sometimes however, CAHPS surveys are implemented for all or almost all health plans in a State or region. The survey(s) used depends on the member populations of interest to the sponsor. CAHPS surveys are intended to be administered by an independent organization, not by the health plans themselves. The goal is to generate data that will be viewed as credible, especially by the public and individuals making a health plan choice.

Sampling and Oversampling Issues

If a routine general population sample is taken of MCO members for purposes of a CAHPS survey, it is likely that the number of people with disabilities in the sample will not be sufficient to permit meaningful conclusions. It will be essential either to target data collection for members who are known (for example by their eligibility category) to have a disability, or to develop methods for "oversampling" the records of people with disabilities.

Risk Adjustment

Items have been included in all CAHPS surveys that gather information on the demographics and general health status of respondents. These data permit basic adjustments for differences between plans in such common risk-adjusters as age and overall self-reported health. Note that these are intended to be adjustments that might affect how people report on their experiences with plans.

Evidence to Support Validity and Reliability of Measures:

CAHPS measures have been developed by nationally recognized experts on surveys of consumers' experiences in health plans and their ratings of health plans. They have been subjected to intensive cognitive testing with the populations for which they are intended, in order to ensure that all questions and response options are understood as intended. Surveys have also been field-tested with relevant populations and psychometric analyses have been conducted to assess their validity and reliability. In addition, surveys have been reviewed by external advisers and experts on such issues as the required educational level for reading surveys (a sixth grade level is the standard for all surveys), and have been subject to public comment. A number of demonstration sites have implemented CAHPS surveys in a context where formal evaluations are being conducted by the CAHPS research teams. Other "early adopters" have used the survey in the course of their normal performance measurement activities. State Medicaid agencies have been among both the demonstration sites and early adopters.

Populations and Settings in Which Measures Have Been/Could Be Used:

CAHPS surveys, as noted above, have been designed with specific populations in mind. All surveys include the "core" items; these items make up the survey of privately insured adults. Other surveys include supplemental item sets designed specifically to address the concerns and needs of particular populations. People eligible for Medicaid and people with chronic illnesses and disabilities were from the outset important survey populations for CAHPS. In addition, HCFA commissioned the development of a CAHPS Medicare HMO survey, which includes many items that are similar to those in the survey of people with chronic illnesses and disabilities. Surveys currently under development, in particular the survey of Children with Special Needs (being developed by Harvard) and the Survey of Medicaid SSI (being developed by RAND) are highly specific to people with disabilities.

One of the major strengths of the CAHPS surveys is that, with the exception of the Medicare survey, they are designed to permit comparisons of member experiences across different kinds of delivery systems. Thus, for example, the survey can be and has been used for people on Medicaid who are in FFS or PCCM systems. As with HEDIS® measures, HCFA has mandated the collection of the CAHPS Medicare HMO survey data for all Medicare risk-contract HMOs serving Medicare beneficiaries (including those eligible because of a disability). This will provide a useful basis for comparisons.

Written Documentation Available to Support Use of Measure:

Extensive and detailed written documentation is available from the Survey Users Network (SUN Network) on each of the fully developed surveys. CAHPS surveys are currently undergoing revisions to reflect the integration of initial CAHPS surveys (Version 1.0) with the NCQA Member Satisfaction Survey. Version 2.0 of CAHPS surveys and a revised version of the "Survey and Reporting Kit" will be available in early 1999. In the meantime, however, written documentation remains available for earlier versions and much of the information is still valid. As with Version 1.0, demonstration sites and other early adopters will have access to the most current version of any fully developed CAHPS survey for implementation, and can get support from SUN.

In addition, CAHPS has developed methods and materials for reporting the results of CAHPS surveys to consumers, including print materials and an interactive computer program. Templates of these materials and documentation on how to adapt them are included in the "Survey and Reporting Kit." These educational materials have gone through the same process of testing and scrutiny as the surveys themselves. Certain reporting products are specific to the Medicaid population.

Consultation Available to Support Use of Measures:

The Survey Users Network contractor can provide consultation on the use of CAHPS surveys that are fully developed, and information on the status of surveys under development. CAHPS research teams that are taking the lead in development of newer surveys would be available for limited consultation on these surveys. In addition to providing technical assistance, the SUN contractor periodically brings together users to share what they have learned about the use of these surveys.

Limits on Use of the Measure:

By definition, surveys of consumers are vehicles for gathering information that the members of plans and users of health care are in a particularly good position to assess and rate. Not all aspects of health plan performance can be assessed and rated by consumers. In addition, there are limits on the use of surveys by individuals who are cognitively impaired or mentally dysfunctional. Finally, while to date the Medicaid surveys have been translated into Spanish, most CAHPS surveys are available only in English. Resources are currently being sought to permit translation of a wider range of the surveys into other languages. It is likely that CAHPS research teams would be willing to work with sites interested in pursuing translation efforts.

4. Oregon Health Plan Adult Satisfaction Survey of 1997

Name of Measurement System: The Oregon Health Plan Adult Satisfaction Survey of 1997

Author/Developer: Oregon Department of Human Resources, Human Resources Building, Office of Medical Assistance Programs, 500 Summer St., NE, Salem, Oregon 97310-1014, Phone (503) 945-6929. Contact: Judy Mohr-Peterson, Evaluation Research Coordinator, Office of Medical Assistance

This is an example of an instrument developed by a State in order to ascertain both the quality and performance of its managed care and fee-for-service models, from the perspective of Medicaid clients. The survey was developed by the Office of Medical Assistance Programs of the Oregon Human Resources Department in Salem, Oregon.

Domains of Performance Addressed by the Measure:

The Oregon Health Plan Adult Satisfaction Survey questions are directed to consumers. It includes a series of headings and sub-headings such as "Your Health Plan," "Thinking About Your Health Plan," and "Thinking About the Doctor's Office or Clinic." We have chosen to include specific queries from several headings.

Domain 1. Creating a System with the Right Capacities

• Ability to have a choice among different plans
• Ease of communication between personal doctor and specialists
• Creating a position of Exceptional Needs Care Coordinator (ENCC) in health plan
• Rating of helpfulness of ENCC and ENCC services

Domain 2. Providing Access to Needed Services

• Ease of access to a particular primary care provider
• Ease of access to specialist doctors
• Means of transportation to primary health care place
• Ease of travel to primary health care location
• Use of interpreter if needed (foreign languages, sign languages)
• Medications packaged for ease of use
• Convenient locations of pharmacies
• Ease or difficulty of early appointments, conveniently timed appointments, appointments with specialists, prescription medicines, medical advice
• Ease or difficulty of obtaining emergency medical care
• Ease or difficulty of maneuvering around medical offices (if blind, visually impaired, or use special equipment to move from place to place)
• Ease of access to counseling/mental health services
• Ease of access to alcohol or drug treatment services
• Ease of obtaining physical, occupational or speech therapy
• Ease of obtaining home health care
• Ease of acquisition of special medical equipment
• Ease of repair of special medical equipment

Domain 4. Resolving Problems and Concerns

• Notification by health plan and/or medical assistance worker of rights to complain or appeal
• Access to information from membership services
• Access to easily understandable information about benefits and services
• Knowledge about Ombudsman staff at the Office of Medical Assistance Programs (OMAP)
• Ease of access to OMAP's Ombudsman staff
• Helpfulness of information received from Ombudsman staff

Domain 5. High Quality Interpersonal Interactions Between Members and Providers

• Providers listen without interrupting or rushing
• Providers explain things clearly
• Providers show respect for patient
• Providers spend sufficient time with patient
• Providers follow through on test results or after care
• Health care professionals do not convey conflicting information
• Providers provide enough information about a health care condition
• Ratings of personal doctor or nurse

Domain 6. Using Preventive Services to Keep Members Healthy and Functioning

• Provider discussion of prevention and maintenance
• Provider reminder of mammogram, pap test, quitting tobacco, cholesterol test, weight control, prostate screening, alcohol or drug screening

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

• Arrangements to "watch kids" at home or provide transportation
• Patient gets help with coordination of medical care needs (from medical provider, from Exceptional Needs Care Coordinator (ENCC) at health plan, from Case Manager at social services, other)
• Rating of helpfulness of provider in giving information about other agencies or services

Domain 8. State of the Art Treatments

• Rating of home health care

Domain 9. Improving the Outcomes of Care

• Patient rating of health status compared to one year previously

Data Collection Strategies Required for Use of the Measure:

Type of Data Collection

Survey forms are mailed by OMAP to Medicaid beneficiaries. The surveys are returned to OMAP and then electronically scanned.

From Whom Are Data Collected

Survey recipients are sampled at random from all Oregon Medicaid beneficiaries who have been enrolled in the Oregon Health Plan for a minimum of six months. The survey is available in Spanish, Russian, and Vietnamese. Potential respondents who speak other languages can use an AT&T translation operator and complete a telephone survey. There is a Client Hot Line to respond to questions from respondents and a prepared list of questions and answers for those answering the hot line telephone.

Sampling and Oversampling Issues

In order to generate a sufficiently large sample of people with disabilities to ensure reliable estimates, this survey, like others presented in the Guide, would either need to be based on a targeted sampling frame, or would require the use of oversampling and/or a special set of screening questions to identify individuals with disabilities.

Risk Adjustment

A limited number of questions in the survey address health status of respondents; they are probably insufficient to support risk adjustment of responses. However, depending upon how the sampling frame for the survey was structured (e.g. by eligibility category) it is possible that responses could be calculated for different groups.

Evidence to Support Validity and Reliability of Measures:

Information not available at this time.

Populations and Settings in Which Measures Have Been/Could Be Used:

The measure has been used with Medicaid and low-income populations who represent a cross-section of eligibility categories, including people with disabilities.

Written Documentation Available to Support Use of Measure:

The survey instrument, as well as sample letters used for mailing the instrument, and scripts used to address Frequently Asked Questions from survey recipients (i.e., beneficiaries of Medicaid and other governmentally support health insurance programs) are available from Judy Mohr-Peterson (see contact information above).

Consultation Available to Support Use of Measures:

Oregon has provided copies of the measure to several other State Medicaid agencies and has provided informal consultation on its use.

Limits on Use of the Measure:

The primary limitation on the measure is that some questions, for example those on Exceptional Needs Care Coordinators, might not be appropriate for settings in which such resources are not required or available. However, it may be possible either to eliminate those questions or replace them with items that are more appropriate to a particular context.

5. Picker Institute Patient Surveys

Name of Measurement System: Picker Institute Surveys of Adult Medical/Surgical Hospital Stays; Rehabilitation Programs; and Home Care Services

Author/Developer: The Picker Institute, 1295 Boylston Street, Suite 100, Boston, MA 02215, Phone (617)667-8484. Contact: Ms. JoAnne Leamey, Marketing Manager

The Picker Institute is a non-profit organization dedicated to increasing the extent to which health care services are patient-centered. They have developed a number of patient surveys that focus on specific aspects of health care, including three that are of particular relevance to the needs of people with disabilities:

• A survey of adult patients who have recently utilized inpatient hospital services in a medical or surgical ward (including questions that address admission through an emergency room if that occurred);
• A survey of patients who have recently utilized a rehabilitation program; and
• A survey of patients who have recently utilized home care services.

Domains of Performance Addressed by the Measure:

The Picker Institute surveys address several domains of performance. We indicate each domain, and give examples of specific items covered in one or more surveys. When an issue is addressed in only one survey, this is so indicated. Note that in the Picker surveys, questions are often directed to the patient's experience with a particular kind of provider; thus several issues are addressed with respect to different providers (e.g., doctors, nurses, social workers, home health aides etc.)

Domain 2. Providing Access to Needed Services

• Timely access to hospital admission from the emergency room (Hospital Survey)
• Availability of health care providers when and as frequently as needed
• Timely performance of needed tests and procedures (Hospital and Rehabilitation Surveys)
• Convenience of timing of home care visits (Home Care Survey)
• Timely availability of pain medication (Hospital Survey)
• Availability of needed special equipment in the home, and changes to home (Rehabilitation and Home Care Surveys)
• Access to sufficient number of home care visits (Home Care Survey)

Domain 3. Supporting Member Involvement in Decision Making and System Improvement

• Patients have enough to say about their treatment

Domain 4. Resolving Problems and Concerns

• Timely response to complaints (Home Care Survey)
• Agency willingness to change home care providers (Home Care Survey)

Domain 5. High Quality Interpersonal Interactions Between Members and Providers

• Providers listen carefully
• Providers explain treatments and answer questions clearly
• Staff and providers are courteous and helpful
• Home care providers are courteous to family and friends (Home Care Survey)
• Staff explain reasons for delays (Hospital Survey)
• Providers treat patients with respect and dignity
• Providers explain risks and benefits of surgery and how patient will feel after surgery (Hospital Survey)
• Providers discuss patient's anxieties and fears and/or offers encouragement
• Providers inspire confidence and trust
• Providers explain test results (Hospital Survey)
• Providers talk to, involve, and provide information to family members
• Providers do not talk about patient in their presence as if they were not there
• Patients get enough privacy (Rehabilitation Survey)

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

• Providers are consistent in what they say to patients (Hospital Survey)
• Presence of one person to coordinate care provided by rehabilitation team (Rehabilitation Survey)
• Support for family involvement in providing care, including care post-discharge (Rehabilitation Survey)
• Arrangements made for post-discharge care from other agencies (Rehabilitation Survey)
• Consistency of provider across time (Home Care Survey)

Domain 8. State of the Art Treatments

• Hospitals provide machinery patients can use to give themselves pain medication (Hospital Survey)
• Staff does all they can to help patients control pain (Hospital and Home Care Surveys)
• Patients are given information on discharge about medications and how to take them, danger signals and other follow-up care (Hospital and Rehabilitation Surveys)
• Patients are given information on nutritional needs, pain control, etc. (Hospital and Rehabilitation Surveys)

Domain 9. Improving the Outcomes of Care

• Patient ratings of provider coordination (Hospital and Rehabilitation Survey)
• Patient ratings of hospital care (Hospital Survey)
• Patient ratings of rehabilitation services (Rehabilitation Survey)
• Patient ratings of home care (Home Care Survey)

Data Collection Strategies Required for Use of the Measure:

Type of Data Collection

The Picker Institute instruments are surveys of patients. They are administered by mail.

From Whom Are Data Collected

Data are collected from individuals who have been identified as having used a particular kind of facility or service, within a specified time period. Data are typically collected by The Picker Institute itself, under contract with health care organizations, MCOs, employers and business or purchaser coalitions.

Sampling and Oversampling Issues

The key sampling issue for Picker Institute Surveys is the ability to identify from records individuals who have used particular services and to contact a random sample from this list. Valid records would need to be provided either by the MCO or by the health care facilities under contract with the MCO. Normally, the surveys would not be conducted only for people with disabilities, although if information were available as to disability status as well as use of services, this could be done. Sample size will, as in all cases, be an issue. On the other hand, it is more likely that a sizable portion of people using rehabilitation and home care services would have some kind of disability.

Risk Adjustment

Information is collected in the surveys on demographic characteristics and health status that could be used for risk adjustment purposes.

Evidence to Support Validity and Reliability of Measures:

Picker Institute measures are developed using a rigorous and extended process. The process typically begins with focus groups of patients who have used the service in question, to identify what is of greatest interest and importance to them. Surveys are usually cognitively tested prior to field testing. Psychometric analyses are conducted on the reliability, validity and discriminate ability of the surveys. As data are collected using a particular survey, these data are analyzed and used to continue psychometric analysis and survey refinement.

Populations and Settings in Which Measures Have Been/Could Be Used:

Picker Institute surveys have been used in a wide range of health care organizations and MCOs. Surveys are not targeted to patients on the basis of their source of insurance, but rather on their use of the facility in question. The surveys presented here appear to be relevant to the needs and concerns of people with disabilities.

Written Documentation Available to Support Use of Measure:

The Adult Inpatient and Rehabilitation surveys are not in the public domain; they are copyrighted and can only be used with permission. The Home Care survey is in the public domain. The Picker Institute has written documentation of the psychometric analyses conducted on their surveys; some articles have also appeared in the academic literature.

Consultation Available to Support Use of Measures:

As noted earlier, The Picker Institute typically administers the surveys themselves for a variety of clients. This would involve a charge to be negotiated with the client.

Limits on Use of the Measure:

The primary limitation on the use of these surveys measure is that two of the three are not in the public domain; however, Picker staff are extremely interested in the widespread use of their surveys and in issues significant to people with disabilities. Translations of the instruments would be required if non-English speaking populations were involved.

V. CRITERIA SETS FOR ASSESSING MANAGED CARE ORGANIZATIONS

This Chapter presents work that has been done to identify structural characteristics and procedures of MCOs that are viewed as highly significant to their capacity to serve the needs of people with disabilities. By themselves, structural characteristics can be seen as criteria for assessing MCOs, rather than as measures. They indicate what should be present, but do not necessarily provide a way to find out if the characteristic is present. The great advantage of structural criteria is that they can be used prospectively, i.e., before an MCO begins to serve people with disabilities. The disadvantage is that the existence of a structure does not guarantee its appropriate use. Structures are necessary, but not sufficient, to the delivery of good care.

Another advantage of the criteria sets in this Chapter is that several were developed with the health care needs and concerns of people with disabilities specifically in mind. The same kind of information on criteria sets is offered as that presented in Chapter Four on measures and measurement systems. For example, criteria are organized using the framework of domains of MCO performance presented in Chapter Three. In many cases, the complete criteria set goes beyond the domains of performance we have laid out in this Resource Guide. Readers are therefore encouraged to go to the complete sets since there may be additional criteria they would want to apply.

How can you use structural criteria sets in a performance measurement system? How can you find out whether a characteristic is in place? There are several options to consider:

• In many cases, structural characteristics and criteria can become contract specifications. Indeed, one criteria set we present was developed as a set of specifications for Medicaid MCO contracts.
• MCOs can be asked to report whether these characteristics are in fact present, and to provide documentary evidence of their presence, such as contracts with providers; provider listings; memoranda of understanding; and procedure manuals.
• Visits can be made and interviews conducted with MCO staff to further probe these characteristics, to answer questions raised by descriptions or documentation, and to gather information from multiple individuals who would be knowledgeable about a given aspect of the MCO's functioning. Such visits and interviews could also be conducted by MCO staff with provider organizations, since MCOs want to know the structural characteristics of those in their network.

Structural criteria often do not specify a clear standard. For example, while an MCO needs sufficient numbers of the right kind of providers in order to serve a patient population, there are few if any research-based standards about how many of what mix of providers is actually sufficient. The Chapter does not suggest such standards. It may be more appropriate for a group of people representing the perspectives of multiple constituencies to work to develop consensus on such standards.

A. Quality Improvement System for Managed Care (QISMC)

The intent of QISMC is to serve as a system for "assuring that managed care organizations contracting with Medicare and Medicaid protect and improve the health and satisfaction of enrolled beneficiaries." HCFA and State Medicaid agencies are expected to use QISMC as a uniform set of quality standards for assessing Medicaid and Medicare contractors. Given the health plan performance and quality requirements embodied in the 1997 Balanced Budget Act (BBA), it is likely that the final version of QISMC will become mandatory for Medicare contracts and optional for state Medicaid contracts. States opting to use QISMC in their Medicaid contracting will likely meet the BBA requirements.

QISMC is described in a 132-page document divided into four parts:

1. A description of a framework for understanding QISMC requirements, including a background on the origins and goals of the project;

2. A list of standards that each organization must meet under QISMC, grouped under four domains:

   • quality assessment and performance improvement;

   • enrollee rights;

   • health services management; and

   • delegation (i.e., scope and applicability of standards as they apply to subcontractors of the contracting health organization);

3. Detailed guidelines for interpreting the standards.

4. The process for implementing QISMIC

Name of Criteria Set: Quality Improvement System For Managed Care (QISMC)

Author/Developer: The National Academy for State Health Policy (NASHP) for the Health Care Financing Administration (HCFA) of the U.S. Department of Health and Human Services

For information regarding the history and development of QISMC:

Alicia Fagan, Director of Special Initiatives, National Academy for State Health Policy, 50 Monument Square, Suite 502, Portland, ME 04101-6422, Phone (207)874-6524, Email info@nashp.org, Internet Website http://www.nashp.org

For information regarding the current status of QISMC:

Brian Agnew, DHHS/HCFA/OA/OCSQ/CSG, Century Building, Room C4-23-16, 7500 Security Blvd., Baltimore, MD 21244-1850, Phone (410)786-0587 Email Bagnew@HCFA.GOV, Internet Website http://www.hcfa.gov/quality/qlty-3e.htm

NASHP is a non-profit multidisciplinary forum of and for State health policy leaders from the executive and legislative branches. Under contract from HCFA, NASHP began development of QISMC in 1996 and released a draft version for public comment in January 1997. Numerous comments were received, and panels of representatives of plans, providers, regulators and advocates were convened. A second round of public comments on the January 1997 draft was initiated by HCFA in May 1998. Based on the comments received on this draft in both rounds, and on its own internal review, HCFA will revise the system and issue it in final form. Readers are advised to contact HCFA at the above address for information regarding the final content of the system.

Domains of Performance Addressed by the Criteria Set:

We have selected the criteria that are most representative of our domains. QISMC contains many more provisions relating to a broad range of health care and delivery issues.

Domain 1. Creating a System with the Right Capacities

• Projects conducted under the organization's QAPI [Quality Assurance/Performance Improvement] program address and achieve improvement in major focus areas of clinical and non-clinical services. Clinical focus areas applicable to Medicaid enrollees include the following:
  • chronic diseases affecting elderly or disabled Medicaid enrollees;
  • care of Medicaid enrollees residing in long-term care facilities;
  • care of Medicaid enrollees who are unusually dependent on others or on devices but who do not reside in long-term care facilities;
  • mental health treatment for Medicaid enrollees;
  • substance abuse treatment for Medicaid enrollees; and
  • care of Medicaid enrollees who are developmentally disabled.
• The organization maintains and monitors a network of providers, supported by written arrangements, that is sufficient to provide timely access to covered services.
  • A new contractor, or an established contractor seeking an expansion of its service area, demonstrates that the numbers and types of providers available to enrollees are sufficient to meet anticipated needs of the population and area to be served.
• For each physician or other individual practitioner, including each practitioner within a contracting group who provides services to the organization's enrollees, the process includes:
  • Formal selection and retention criteria that do not discriminate against practitioners who serve high-risk populations or who specialize in the treatment of costly conditions.

Domain 2. Providing Access to Needed Services

• The organization assures that services are accessible to all enrollees, including those with limited English proficiency or reading skills, with diverse cultural and ethnic backgrounds, and with physical and mental disabilities.
• Enrollee information is
  • readable and easily understood;
  • available in the language(s) of the major population groups served and, as needed, in alternative formats for the visually impaired.
• An established contractor establishes standards for timeliness of access to care and member services that meet or exceed such standards as may be established by HCFA or the State Medicaid agency, continuously monitors the extent to which it meets these standards, and takes corrective action as necessary.

Domain 3. Supporting Member Involvement in Decision Making and System Improvement

• The QAPI program provides opportunities for enrollees to participate in the selection ofproject topics and the formulation of project goals.
• Each enrollee has a right
  • to choose providers from among those affiliated with the organization
    • Each enrollee may select his or her primary care provider from among those accepting new Medicare or Medicaid enrollees;
    • Each enrollee may refuse care from specific practitioners or providers.
  • to participate in decision-making regarding his or her health care
    • The organization provides for the enrollee's representative to facilitate care or treatment decisions when the enrollee is unable to do so.
    • The organization provides for enrollee or representative involvement in decisions to withhold resuscitative service, or to forgo or withdraw life-sustaining treatment, and complies with requirements of Federal and State law with respect to advance directives.
  • to receive information on available treatment options or alternative courses of care;
  • to have access to his or her medical records in accordance with applicable Federal and State laws

Domain 4. Resolving Problems and Concerns

• Each enrollee has a right
  • to obtain a prompt resolution, through established procedures, of issues raised by the enrollee, including complaints or grievances and issues relating to authorization, coverage, or payment of services.
• The organization has a system for resolving issues raised by enrollees, including complaints or grievances; issues relating to authorization of, coverage of, or payment for services; and, in the case of Medicare enrollees, issues relating to a discontinuation of a service. [NOTE: references to an enrollee in these standards include reference to an enrollee's representative.]

Domain 6. Using Preventive Services to Keep Members Healthy and Functioning

• The organization conducts a quality assessment and performance improvement (QAPI) that achieves, through ongoing measurement and intervention, demonstrable and sustained improvement in significant aspects of clinical care and non-clinical services that can be expected to affect enrollee health status, functional status, and satisfaction.
• [QAPI includes] measures to assure that enrollees are informed of specific health care needs that require follow-up and receive, as appropriate, training in self-care and other measures enrollees may take to promote their own health.

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

• The organization promotes continuity of care and integration of services through:
  • Programs for coordination of care, including:
    • Identification of enrollees with complex needs and development of services and programs to assist them in meeting those needs;
    • Coordination of medical care, mental health services and substance abuse services, and social services and community resources.

Domain 8. State of the Art Treatments

• The organization adopts and disseminates practice guidelines or criteria for the provision of specific services.
  • Guidelines are based on reasonable medical evidence or a consensus of relevant practitioners, are developed in consultation with affiliated providers, and are reviewed and updated periodically.
• The organization implements written policies and procedures for evaluating new medical technologies and new uses of existing technologies.

Evidence to Support Validity and Reliability of the Criteria Set:

These standards were developed by NASHP utilizing its recognized expertise in analyzing the myriad roles and responsibilities of States as purchasers, regulators, and partners of health plans, particularly as these roles have become increasingly complex with the growth of managed care. A broad-based consensus development process that incorporated the perspectives of diverse stakeholders was employed to bring both depth and quality to QISMC. HCFA is continuing this approach as each of the standards are "vetted" during its own public comment and review period. The fact that HCFA intends for QISMC to become a required feature of Medicare contracting and that the agency highly recommends it as an option for State Medicaid contracting is evidence of its confidence in the utility and reliability of the system.

Populations and Settings in Which Criteria Set Has Been/Could Be Used:

QISMC was developed to apply across the needs of all Medicare and Medicaid managed care enrollees.

Written Documentation Available to Support Use of Criteria Set:

The January 1997 version of QISMC and a summary of public comments and proceedings from the April 1998 meeting of the Quality of Care panel are available free of charge as WordPerfect 6.1© self-extracting zip file at HCFA's website, the address of which is http://www.hcfa.gov/quality/qlty-3e.htm. When a final version of QISMC is released by HCFA, it will also be available on the website and through the contact person identified above.

Consultation Available to Support Use of Criteria Set:

NASHP staff are available to provide consultation and technical assistance to State-level government officials regarding the history and development of QISMC free of charge. Other users are advised to contact NASHP regarding their policies on rates for consulting. Inquiries to HCFA should be directed to Brian Agnew of that office for appropriate guidance on the scope and cost of consultation and technical assistance.

Limits on Use of the Criteria Set:

Among the comments received during the April 1998 Quality of Care meeting were: (1) "Plans and some other commenters noted that plans can't identify some of the populations referred to, such as the developmentally disabled or persons dependent on assistance or devices; even encounter data might not identify these enrollees;" and (2) "Advocates wanted stronger requirements in [the area of coordination]. There was agreement to add 'identification and assessment' to the standard [3.2.2]. There was a wider consensus that guidelines should reflect the need for screening enrollees to determine who has complex needs and may require an assessment, and to better define the term 'complex needs.'"8 HCFA has taken these, as well as all other comments, under advisement as it completes the final version of QISMC. Readers interested in how it may ultimately reflect the needs of people with disabilities should contact HCFA for the most up-to-date information.

B. Key Indicators from the 1997 Santa Fe Summit on Behavioral Health

In 1997 the American College of Mental Health Administration (ACMHA) held a "Summit" in Santa Fe, New Mexico as the final process of consensus development on core values and indicators that should guide the assessment of behavioral health programs. Their final report presents these core values and indicators, and also, unlike other criteria sets in this Chapter, presents information on potential approaches to measurement for these indicators. Since measurement strategies have not been fully developed for all indicators, however, it is more appropriate at this time to present them as a criteria set. Work is ongoing, however, to move these indicators closer to the measurement process.

Name of Criteria Set: Key Indicators from the 1997 Santa Fe Summit on Behavioral Health

Author/Developer: The indicators are clearly the result of the work of multiple individuals and their organizations. However, the final report was produced by The American College of Mental Health Administrators, 7625 West Hutchinson Avenue, Pittsburgh, PA 15218-1248, Phone (412) 244-0670. Contact: Lawrence Heller, Ph.D., Executive Director

Domains of Performance Addressed by the Criteria Set:

The Summit Final Report categorizes its indicators into four domains: Outcomes, Process, Access and Structure. We have reorganized these into the domains used in the framework presented in this Guide.9

Domain 1. Creating a System with the Right Capacities

• The organization's structure is consistent with the delivery of mental and addictive disorder treatment, with effective consumer and professional representation in policy making.
• Staffing levels are appropriate for delivery of the array of services and provide for meeting the diverse needs of the individuals served, including linguistic and cultural competence.
• Data on clients are secure, available only to those who need to know.
• There is a single, fixed point of responsibility for each client.
• There is a quality assurance system in place to examine adverse clinical events.

Domain 2. Providing Access to Needed Services

• Service denials, terminations or refusals are assessed.
• Penetration (i.e. utilization) rates demonstrate benchmarked levels of service delivery to like populations.
• Access to a full range of services is demonstrable.
• Children and their families receive the appropriate services that they need, when they need them.
• Consumer experiences of treatment (both positive and negative) are assessed on dimensions of appropriateness, timeliness and sensitivity of services delivered.

Domain 3. Supporting Member Involvement in Decision Making and System Improvement

• Consumers actively participate in decisions concerning their treatment.
• Consumers receive inpatient mental health services in a voluntary, non-coercive manner.
• Consumers receive adequate information to make informed choices.
• Consumers and their families are educated about their rights, the array of services available to them and likely outcomes of treatment interventions.

Domain 4. Resolving Problems and Concerns

• Consumer rights are clearly defined and procedures for resolution of complaints and grievances are in place and easy to use.

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

• The system of care assumes responsibility for continuous and integrated care appropriate to the needs of families and children.
• There are appropriate linkages to other service systems with which consumers need to interact.
• There is continuity of care within the organization and effective integration with external caregiving systems.

Domain 8. State of the Art Treatments

• Consumers who use inpatient services (i.e., 24-hour, medically-supervised services for a primary mental or substance abuse diagnosis) receive face-to-face follow-up care within seven days of discharge.
• Consumers with mental health and addictive disorders are engaged in treatment (i.e., continue in treatment).
• Children and their families are being assessed for and offered services at appropriate levels (i.e., with respect to use of out-of-home or restrictive environments).

Domain 9. Improving the Outcomes of Care

The Summit indicators are especially useful with respect to their specification of outcome criteria. They have developed three for all adults, including those with serious and persistent disorders, and five relevant only for adults with serious and persistent disorders:

Indicators for All Adults

• Adults reside in their own homes or living arrangements of their own choosing.
• Adults are working (defined to include education, job training, volunteer work).
• Adults have good physical health and report good mental health (psychological well-being).

Indicators for Adults with Serious and Persistent Disorders Only

• Adults with serious and persistent mental or chemical dependency disorders report feeling safe.
• Adults with serious and persistent mental or chemical dependency disorders avoid trouble with the law.
• Adults with serious and persistent mental or chemical dependency disorders maintain a social support network.
• Adults with serious and persistent mental or chemical dependency disorders are able to manage their daily lives.
• Adults with serious and persistent mental or chemical dependency disorders report a positive quality of life.

Evidence to Support Validity and Reliability of the Criteria Set:

The Summit Indicators were developed through extended discussions of multiple working groups. Participants in these groups included mental health and substance abuse professionals working in both private and public sector organizations; academic experts in these field; and some representatives of people with mental health and substance abuse diagnoses. The process used is similar to the consensus development process often used within the National Institutes of Health with respect to the appropriateness of a particular approach to a medical intervention, with this difference, that the values and expectations of participants in the process, as well as their expertise and experience, are brought to bear in the discussions.

Populations and Settings in Which Criteria Set Has Been/Could Be Used:

The criteria were designed for use in assessing the performance of behavioral health care organizations, and could be applied to the performance of MCOs providing behavioral health services to their members.

Written Documentation Available to Support Use of Criteria Set:

The complete Final Report on the Santa Fe Summit, entitled Preserving Quality and Value in the Managed Care Equation is available from ACMHA at the address noted above. Additional materials which go into greater detail on indicators and on measures related to these indicators may also be available from the organization and its collaborators.

Consultation Available to Support Use of Criteria Set:

ACMHA indicates that it would be available to discuss the report with interested parties.

Limits on Use of the Criteria Set:

Some of the indicators are at a very high level of generality; others are more specific and would be easier to use as a basis of assessments. It is important to note, however, that this effort is ongoing, in particular with respect to the dimension of identifying and coming to consensus on appropriate measures for the indicators.

C. National Health Law Program (NHeLP) Advocacy Checklist for People With Disabilities

The NheLP Advocacy Checklist was originally developed as a guide for advocates to use in reviewing the specifications of contracts between State Medicaid agencies and MCOs. While contract specifications are one approach to determine whether or not structural characteristics are in place, the NHeLP Advocacy Checklist can be viewed as a set of criteria that can be assessed using other methods besides inclusion in contracts.

Name of Criteria Set: National Health Law Program (NHeLP) Advocacy Checklist for People with Disabilities

Author/Developer: National Health Law Program (Main Los Angeles Office), 2639 South La Cienaga Blvd., Los Angeles, CA 90034, Phone (310)204-6010, E-mail nhelp@healthlaw.org, Internet Website http://www.healthlaw.org/.

The National Health Law Program, with offices in Los Angeles and Washington, D.C., is a non-profit organization that has served as a resource center for legal services programs and other advocates for the health care needs of low-income people. The Advocacy Checklist was originally developed, as its title suggests, for use by advocates for people with disabilities. However, the criteria can be used not only by such individuals and organizations, but by State Medicaid agencies, MCOs and health care organizations and professionals.

Domains of Performance Addressed by the Criteria Set:

The NHeLP Advocacy Checklist includes criteria that go well beyond the domains of performance included in this Resource Guide. For example, they address marketing issues, the use of third-party organizations to enroll Medicaid eligibles in MCOs, public disclosure issues, etc. We present only those criteria and standards that are relevant to the domains of performance we have articulated. Readers are advised to review the entire checklist.

Note that in some cases a criterion incorporates a specific standard, e.g., that face-to-face health assessments be provided with 60 days of enrollment.

Domain 1. Creating a System with the Right Capacities

• MCO is accredited by NCQA
• MCO adheres to reporting requirements of HEDIS® 3.0
• MCO implements a quality assurance and improvement plan
• MCO uses HCFA QARI quality assurance measures
• In building its network, MCO utilizes maximum patient to FTE primary care ratios that take into account physician participation in multiple networks and with commercial patients
• MCO has providers available, including pediatric specialists, who are no more than 20 minutes away from members in urban areas and 30 minutes away from members in rural areas
• MCO ensures that members whose primary language is not English and members with special medical needs have access to primary care providers and specialists qualified to meet their needs
• MCO takes appropriate responsibility for providing medical transportation (in accordance with statute)
• MCO uses definition of medical necessity consistent with statute and Medicaid and EPSDT regulations
• Definition of medical necessity for behavioral health services is consistent with Federal and State law and recognizes the role of the member/family, least restrictive treatment setting and wraparound services

Domain 2. Providing Access to Needed Services

• For persons with disabilities, chronic and/or complex conditions, ability to choose a specialist as a primary care provider
• For children with disabilities, chronic and/or complex conditions, ability to choose a pediatric subspecialist as a primary care provider
• Where the MCO network does not include an appropriate specialist, or if there are fewer than two specialists or sub-specialists of a certain type, access to out-of-network specialists
• Availability of "standing referrals" to specialists for individuals with ongoing treatment needs
• MCO guarantees 24-hour, seven-day-a-week access to qualified providers
• Appointments for routine care available within ten days; appointments for specialty care available within three weeks; appointments for urgent care available within 24 hours
• Emergency care available at nearest facility, even if not in network
• "Prudent lay person" definition of appropriate use of emergency room services is used
• MCO specifies maximum in-office waiting times
• MCO provides access to clinical studies
• A simple process is in place to obtain prescription medications not included in the formulary

Domain 3. Supporting Member Involvement in Decision Making and System Improvement

• MCO provides information on available providers and member rights and responsibilities, including information on selecting a primary care provider
• MCO provides member information (verbally and in writing) at a reading level set by the State, and in the recipient's primary language, and in alternative formats including TTY and telecommunications devices, Braille, large print and cassette
• Each eligible person can choose his or her own primary care provider from among the MCO's participating providers
• Members can change primary care providers with cause at any time
• Definition of medical necessity is clearly spelled out in contracts and sub-contracts
• Definition provides that treating physician determines medical necessity
• MCO permits second opinions when treating physician determines care is not medically necessary for a person with a chronic or disabling condition or disease
• Consumers are involved in "readiness reviews" of MCOs
• MCO includes consumers in work groups, advisory boards or other "accountability loops"

Domain 4. Resolving Problems and Concerns

• Description of member due process rights is posted conspicuously in all provider locations
• MCO informs members on how to obtain assistance in filing a grievance and of potential availability of free legal services
• MCO informs members of timeframes for plan grievance procedures, State fair hearing and expedited reviews
• MCO informs members of their right to a State fair hearing without exhausting MCO grievance procedures
• Timeframe for MCO grievance procedure is less than 30 days
• Expedited review process is within 48 hours
• MCO provides members with notice when services are denied, reduced or terminated, along with reasons for the action taken and the right to continued services pending final decisions
• MCO has consumer relations office for member questions, problems and complaints

Domain 6. Using Preventive Services to Keep Members Healthy and Functioning

• MCO provides EPSDT services in accordance with Federal regulations and national professional standards of care

Domain 7. Coordinating and Integrating Medical and Non-Medical Services

• Procedures in place to ensure continuity of care if the member's primary care provider leaves the network
• Procedures in place to ease transition into managed care by allowing people with disabilities to retain current providers for a period of time
• MCO contracts with or develops coordination and referral agreements with agencies providing related services
• MCO provides case management to facilitate necessary medical, educational, social and other services

Domain 8. State of the Art Treatments

• MCOs honor ongoing plans of care initiated prior to enrollment until enrollee is evaluated by new primary care provider and a new plan of care is established
• MCO provides face-to-face initial health assessment for all new members within 60 days of enrollment

Evidence to Support Validity and Reliability of the Criteria Set:

These criteria are based on NHeLP's experience in providing advocacy to people with disabilities, as well as other people on Medicaid. In some cases, the criteria and standards are based on research evidence, but in most cases the criteria have what could be called "face validity," i.e. those with concerns about how health services are provided to persons with disabilities believe that the conditions are necessary to the delivery of good care. It is important to note that people concerned about people with disabilities may disagree about the appropriateness of certain criteria. These criteria reflect, as do all performance measures, values and expectations, which may differ from person to person.

Populations and Settings in Which Criteria Set Has Been/Could Be Used:

This criteria set was developed specifically to address the needs of Medicaid eligible people with disabilities, including both adults and children.

Written Documentation Available to Support Use of Criteria Set:

The complete Advocacy Checklist is available from NHeLP's Website, the address of which is http://www.healthlaw.org/checklist-disabilities.

Consultation Available to Support Use of Criteria Set:

NheLP staff would be available to discuss the checklist and its development, but are not experts in the translation of criteria into formal measures.

Limits on Use of the Criteria Set:

As noted above, these criteria do not represent fully developed measures. Additional work is needed to develop methods for collecting data to determine if the criteria are being met. In addition, many criteria represent necessary but not sufficient conditions for high performance, and must be supplemented by the use of other performance measures. Finally, work will be needed to develop consensus on which criteria are appropriate in a given context, and where necessary to specify standards as well as criteria.

NOTES

1. Indeed, this accelerated attention has begun to come full circle: many are now demanding approaches to accountability that level the playing field, by measuring the performance not only of MCOs but of FFS and PCCM systems as well.

2. Shoshanna Sofaer, Dr.P.H., served as the Principal Investigator for this project. She was formerly the Director of The Center for Health Outcomes Improvement Research, which closed on July 1, 1998. Dr. Sofaer coordinated final production of the Resource Guide under the auspices of the George Washington University's Center for Health Policy Research. Inquiries to the George Washington University should be directed to D. Richard Mauery, MPH at (202)530-2376. Dr. Sofaer is currently the Schering-Plough Professor of Health Policy at the School of Public Affairs at Baruch College, New York, where she can be contacted at (212)802-5980.

3. At this point, policymakers at higher levels than the State agency, as well as legislators, may become involved.

4. This requires that the number of persons served by provider groups is large enough to permit such comparisons.

5. Some of the measures included in the Resource Guide come from the HEDIS® measurement system of the National Committee on Quality Assurance (NCQA). NCQA has begun to identify and provide standardized training to organizations it believes are qualified to conduct HEDIS® measures.

6. The Appendix presents an example of the specific issues faced in collecting performance data using administrative information alone.

7. HEDIS® 3.0 included a Member Satisfaction Survey. We have not included measures from that Survey, since in a recent decision, NCQA decided to collaborate with the Consumer Assessment of Health Plans (CAHPS) project to develop a single survey of health plan members. The version of the CAHPS survey which has grown out of this collaboration is separately presented in the part 3 of this section.

8. "Quality Improvement in Managed Care: Summary of Key Public Comments and Quality of Care Group Recommendations". NASHP, April 7, 1998. Available at http://www.nashp.org/pubs/qlty0001.htm. Accessed July 1, 1998.

9. The report also includes a discussion of outcomes relevant to children and adolescents with serious emotional disorders. These outcomes were developed after the Santa Fe Summit but mirror its structure and underlying values. They are included in the Final Report cited here.

REFERENCES AND RESOURCES

American College of Mental Health Administration. 1997. The Santa Fe Summit on Behavioral Health: Preserving Quality and Value in the Managed Care Evaluation, Final Report. Pittsburgh, Pennsylvania: American College of Mental Health Administration.

American Federation of Labor and Congress of Industrial Organizations. 1997. Union Guide to Quality Managed Care. Washington, D.C.: AFL-CIO.

American Managed Behavioral Healthcare Association. 1998. Website: http://www.ambha.org. Accessed May 19, 1998.

American Managed Behavioral Healthcare Association. 1996. PERMS: A Work-In-Progress Status Report. Washington, D.C.: AMBHA.

American Managed Behavioral Healthcare Association Committee on Quality Improvement and Clinical Services. 1998. Perms 2.0: Performance Measures for Managed Behavioral Healthcare Programs. Washington, D.C.: AMBHA.

Booth, M. Look Before You Leap: Assuring the Quality of Care of Managed Care Programs Serving Older Persons and Persons with Disabilities. 1996. Portland, Maine: National Academy for State Health Policy.

CAHPS Consortium. 1997. Consumer Assessment of Health Plans Surveys. Rockville, Maryland.

Coursey, R.D. 1997. CMHS Managed Care Initiative: Adult Panel. Summary of Basic Findings, Gaps, and Future Directions in Adult Practice Guidelines, Standards, and Provider Competencies. Rockville, Maryland: Center for Mental Health Studies.

Davis, M.H. and O'Brien, E. 1996. DataView: Profile of Persons with Disabilities in Medicare and Medicaid. Health Care Financing Review, vol. 17, no. 4, 179-215.

Durch, J.S., Bailey, L.A., and Stoto, M.A. eds. Summary: Improving Health in the Community. A Role for Performance Monitoring. Washington, D.C.: Institute of Medicine National Academy Press.

Felt-Lisk, S., St. Peter, R. 1996. The Quality Assurance Reform Initiative (QARI) Demonstration for Medicaid Managed Care Final Evaluation Report. Washington, D.C.: Mathematica Policy Research, Inc.

Gold, M., Sparer, M., and Chu, K. 1996. Medicaid Managed Care: Lessons From Five States. Health Affairs, vol. 15, no. 3, 153-166.

Goldman, S.K. and De La Cruz Irvine, M. 1997. Report of the Child and Adolescent Panel for the Mental Health Managed Care and Workforce Training Project. Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, and The Center for Mental Health Health Policy and Services Research at the University of Pennsylvania Medical Center.

Gordon, P. 1997. Managed Care for Medicaid Clients with Disabilties: Colorado's Participatory Approach to Implementation. Washington, D.C.: U.S. Department of Health Care Policy and Financing.

Gore, M.J. 1998. Measurement Initiatives Multiply as Industry Matures. 1998 Comparative Performance Data Sourcebook. New York: Faulkner and Gray, 389-395.

Hall L.L., Edgar E.R., Flynn L.M. 1997. Stand and Deliver: Action Call to a Failing Industry. The NAMI Report Card. Arlington, Virginia: National Alliance for the Mentally Ill.

Harahan, M. 1997. Defining the Populations. Slide presentation based on data from the 1994 National Health Interview Survey to the AHCPR User Liaison Program Meeting on Managed Care and Persons with Disabilities and Chronic Ilness, in Phoenix, Arizona, November 5, 1997.

Health Care Financing Administration. 1998. April 1998 Draft: Key Approaches to the Use of Managed Care Systems for Persons with Special Health Care Needs. Website: http:://www.hcfa.gov/medicaid/smd-snp.htm. Accessed May 26, 1998.

Health Care Financing Administration Medicaid Managed Care Technical Advisory Group and The Center for Vulnerable Populations. 1996. A Framework for the Development of Managed Care Contracting Specifications for Dually Eligible Adults. Portland, Maine: The National Academy for State Health Policy.

Health Care Financing Administration. 1997. Quality Improvement System for Managed Care (QISMC). Website: http://www.hcfa.gov/quality/qlty-3e.htm

Hill, S., Cybulski, K., and Thornton, C. 1997. Draft Survey Questionnaire for a Study of Disabled SSI Beneficiaries in TennCare. Princeton, New Jersey: Mathematica Policy Research.

Institute on Disability at the University of New Hampshire. 1998. MCARE: Charting the Changing Tides of Managed Care and Long-Term Services. Website: http://www.mcare.net/brochure.html. Accessed March 19, 1998.

Jencks, S.F. 1995. Measuring Quality of Care Under Medicare and Medicaid. Health Care Financing Review, vol. 16, no. 4, 39-54.

Kastner, T.A. and Walsh, K.K. 1997. Measuring and Improving Quality in Managed Health Care for People with Developmental Disabilties. Developmental Disabilities Health Alliance, Inc. for the Center for Health Care Strategies, Inc.

Leginski, W.A., Croze, C., Driggers, J. et al. 1989. Mental Health Service System Reports: Data Standards for Mental Health Decision Support Systems. Washington, D.C.: United States Department of Health and Human Services, Public Health Service, Alcohol, Drug Abuse, and Mental Health Administration, National Institute of Mental Health.

Manderscheid, R.W., Sonnenschein, M.A., eds. 1996. Mental Health, United States, 1996. Washington, D.C.: U.S. Department of Health and Human Services, Public Health Service, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services.

McNeil, J.M. 1993. Americans with Disabilities: 1991-92. U.S. Bureau of the Census Current Population Report. Washington, D.C., pages 70-73.

Miller, R.H. and Luft, H.S. 1997. Does Managed Care Lead to Better or Worse Quality of Care? Health Affairs, vol. 16, no. 5, 7-25.

National Academy for State Health Policy as part of the Quality Improvement System for Managed Care (QISMC) Initiative. 1997. Public Review Draft: Standards and Guidelines for Review of Medicare and Medicaid Managed Care Organizations. Washington, D.C.: Health Care Financing Administration.

National Committee for Quality Assurance. 1997. Health Plan Employer Data and Information Set, HEDIS® 3.0/1998. Washington, D.C.

National Health Law Program (NHeLP), Inc. 1998. Medicaid Managed Care Contracts: An Advocacy Checklist for People with Disabilities. Website: http://www.healthlaw.org/checklist-disabilites.html. Accessed March 19, 1998.

Newman, S.J., and Envall, K. 1995. The Effects of Supports on Sustaining Older Disabled Persons in the Community. Washington, D.C.: American Association of Retired Persons.

Office of the Assistant Secretary for Planning and Evaluation, Health Care Financing Administration, and the American Association of Retired Persons. 1996. Beyond the Water's Edge: Charting the Course of Managed Care for People with Disabilities. Conference materials from the meeting in St. Michaels, Maryland, November 20-22, 1996.

Oregon Department of Human Resources Office of Medical Assistance Programs. 1997. Oregon Health Plan Adult Satisfaction Survey. Salem, Oregon.

Perkins, J. 1998. Managed Care Update. Los Angeles, California: National Health Law Program, Inc.

Picker Institute. 1995. Surveys of Adult Medical/ Surgical Hospital Stays; Rehabilitation Programs; and Home Care Services. Boston, Massachusetts: The Picker Institute.

Rosenbach, M.L. 1995. Access and Satisfaction Within the Disabled Medicare Population. Health Care Financing Review, vol. 17, no. 2, 147-167.

Segre, G., Loughran, K., Rosenthal, B. eds. 1998. 1998 Comparative Performance Data Sourcebook. New York: Faulkner and Gray.

Social Security Administration. 1998. SSA Handbook. Section 601. Website: http://www.ssa.gov. Accessed May 5.

Sofaer, S. 1995. Performance Indicators: A Commentary From the Perspective of an Expanded View of Health. The Center for the Advancement of Health and The Western Consortium for Public Health.

United Cerebral Palsy Associations, Inc. 1997. Report on the Development of Indicators to Monitor the Performance of Systems of Acute Health Care for Children with Disabilities. Washington, D.C.: U.S. Departments of Education and Health and Human Services.

Van Campen, C., Sixma, H., Freile, R.D., et al. 1995. Quality of Care and Patient Satisfaction: A Review of Measuring Instruments. Medical Care Research and Review, vol. 52, 109-133.

APPENDIX: COLLECTING DATA FOR A PARTICULAR PERFORMANCE MEASURE: AN EXAMPLE

This example illustrates in some detail the process, resources and issues in collecting data for a particular performance measure, the HEDIS® 3.0 childhood immunization measure. These data could only be produced from claims/encounter data if:

• All children who turn 24-months-of-age during the reporting period were continuously enrolled from date of birth through their second birthday; AND
• All children received their immunizations from their MCO primary care physician or another MCO practitioner; AND
• The MCO practitioner correctly coded the specific immunization given on the claim/encounter form, regardless of whether the practitioner was directly compensated for the immunization and the serum.

However, recognizing that Medicaid children may have short periods of enrollment, the HEDIS® 3.0 specifications require that for a Medicaid child to be included in the at-risk population, the child need only be enrolled for 11 out of 12 months in the second year of life. Thus, the child may not have received any immunizations through the MCO in the first year of life and the MCO may not have claims/encounter data for these services. Additionally, even while enrolled in the MCO, the child may have received immunizations through other providers, such as the health department, at immunization health fairs, in a Head Start program, etc. Again, the MCO would not have claims/encounter data. Lastly, the MCO may have difficulty getting accurate, detailed claims/encounter information from its practitioners if it does not reimburse its practitioners on a fee-for-service basis for these services. For example, if practitioners receive free vaccine through the health department, they only bill the MCO for administration of the vaccine and may not be concerned about using the correct vaccine code when they are receiving a standard fee for administration of the vaccine. Similarly, primary care physicians who are paid on a capitation basis may not be highly motivated to provide timely and accurate immunization encounter data since the data are not directly related to reimbursement.

Consequently, almost all MCOs that report the HEDIS® 3.0 childhood immunization measures must use a combination of claims/encounter data and review of medical records to collect the data. Medical record review is costly because it is labor-intensive, requires knowledgeable data abstractors, and involves data entry. In sum, collecting data on performance measures is dependent on how MCOs are organized, and on the sophistication and reach of their management and clinical information systems. States may require, through contract specifications for example, that specific data be provided by MCOs in a certain form. However, it is important that all participants in performance measurement systems be realistic about whether resources are available to collect data that inspire confidence.