Developing Quality Measures for Medicaid Beneficiaries with Schizophrenia: Final Report

Executive Summary

Sam Simon, Thomas Croghan, Robert C. Saunders, Sarah Hudson Scholle, Milesh M. Patel, and Jeremy Gottlich

Mathematica Policy Research

January 30, 2012

In August 2010, the U.S. Department of Health and Human Services Office of the Assistant Secretary for Planning and Evaluation (ASPE) contracted with Mathematica Policy Research and its subcontractor--the National Committee for Quality Assurance--to develop evidence-based quality measures to assess the quality of care provided to Medicaid enrollees diagnosed with schizophrenia. The goal of the project was to create a set of claims-based ambulatory care measures that meet National Quality Forum (NQF) criteria for importance, scientific acceptability, usability, and feasibility and would thus be suitable for submission to the NQF for endorsement consideration.

The project began with a review of existing literature and other evidence describing evidence-based practices for people with schizophrenia. Assisted by expert consultants, this effort emphasized the findings of the Schizophrenia Patient Outcomes Research Team and allowed the team to create concepts for new measures that assess the quality of medication management, underuse of evidence-based psychosocial treatments, and access to primary care and preventive health services. Once the measure concepts were vetted by a Technical Advisory Group (TAG), we developed draft specifications and sought comment from measure stakeholders, including representatives from managed behavioral healthcare organizations (MBHOs), Medicaid medical directors, and state mental health directors to assess their perspectives on the importance, scientific acceptability, usability, and feasibility of the proposed measures. After these key stakeholders gave their input, measure specifications were posted for public comment, and they were pilot-tested using Medicaid Analytic eXtract (MAX) data from 2007 and 2008 to further assess their feasibility, reliability, and validity. Throughout the project, the project team received valuable advice and guidance from ASPE, members of the TAG, and our project consultants.

The project team sought to develop measures in three domains, pharmacology, psychosocial care, and physical health, as well as cross-cutting measures that span several of these domains. Based on the review of the literature and feedback from the TAG and ASPE, we developed detailed specifications for an initial set of 17 measure concepts before settling on a final set of ten to be submitted to NQF for endorsement.

Focus groups with state Medicaid and mental health leaders, as well as with MBHO staff, yielded remarkably consistent results. Key points included: (1) claims data are unreliable for identifying some behavioral health services, particularly evidence-based psychosocial treatments; (2) variation in financing of services for people with serious mental illness (SMI) limits the ability to consistently measure the quality of care across Medicaid programs; and (3) some candidate measures address problems that are not unique to patients with schizophrenia--measures could be broadened to include patients with bipolar disorder, schizophrenia, and severe forms of depression. The feedback from public comment was positive, with 87 percent of the comments either supporting the measures or supporting them with modifications.

Overall, 9.7 percent of Medicaid recipients in our 22-state 2007 MAX dataset had schizophrenia and 12.8 percent had SMI (bipolar disorder and/or schizophrenia). The objective of pilot-testing was to determine the scientific acceptability of each measure to the extent practicable through the use of Medicaid claims data. Five of the ten proposed measures demonstrated significant variability in state-level performance, indicating general utility of the measures. Seven of the ten proposed measures demonstrated evidence of either construct or convergent validity. Construct validity was assessed by examining the association between measure performance and outcomes (schizophrenia-related (1) hospitalization, and (2) emergency department [ED] visits). We reported the percentage of people who were either hospitalized or visited the ED for schizophrenia, comparing the worst and best-performing quartiles of state performance for each measure. Seven measures demonstrated evidence of construct validity, indicated by the association between (higher) measure performance and (lower) rates of adverse events. Convergent validity was determined through enrollee-level measure correlations. Three of the ten measures demonstrated evidence of convergent validity. Nine of the ten measures demonstrated evidence of reliability, assessed between measures calculated during calendar year 2007 and 2008, either through test-retest correlations or relative performance stability over this time period.

Although some of these results are encouraging, important limitations of our findings warrant consideration. First, use of Medicaid claims data as a source to implement and test schizophrenia quality measures limited the number of evidence-based practices that could be implemented as measures. This limitation prevented our ability to develop psychosocial measures. In addition, several topics could not be developed because the evidence base, tools, and methods for tracking these measures are immature. We also found that variation in the financing of services for people with SMI limited our ability to generalize measurement of the care provided by Medicaid programs. For example, the provision of services through state mental health systems, the coverage of mental health services through Medicare for dual-eligible beneficiaries, the prohibition of same-day billing of medical and behavioral health services, and interstate variation in Medicaid and disability standards all underscore the limitations of claims data to measure quality for enrollees with schizophrenia. Finally, the distinction between enrollees with schizophrenia and other SMI conditions is, in many cases, artificial. The project team, ASPE, and measure stakeholders all expressed the belief that conceptually, many issues related to schizophrenia also apply broadly to people with any SMI. Further work is needed to consider whether measures similar to the ones developed and tested under this contract would be relevant for people with bipolar disorder and other SMI.