Developing Quality Measures for Medicaid Beneficiaries with Schizophrenia: Final Report. IV. Lessons Learned


While we successfully developed and tested ten quality measures, development of several additional measures was not feasible given the constraints of Medicaid claims data and Medicaid payment policies. The following discussion of our experience and lessons learned is designed to be instructive for future efforts in the development of quality measures for people with SPMI.

  1. Use of Medicaid claims data as a source to implement and test schizophrenia quality measures presented several noteworthy limitations. Because of the limitations of the claims data, several evidence-based practices could not be implemented as measures. These limitations were particularly conspicuous when attempting to operationalize evidence-based guidelines for psychosocial treatments such as those recommended in the Schizophrenia PORT. In analyses using MAX data, we found psychosocial treatments are either inconsistently coded in claims data or not available at all. For example, claims for smoking cessation programs were not observed in the MAX data; therefore, this measure was not developed because it could not be assessed in claims data. Consequently, no psychosocial measures emerged from our measure development process, despite the strength of evidence for these practices. Specific evidence-based recommendations that could not be accurately identified in the claims data, and thus were not field or pilot-tested, included:

    • Supported employment;
    • Family psychoeducation;
    • Assertive community-based treatment;
    • Cognitive behavioral therapy;
    • Social skills training.

    Claims-only assessment presents other challenges for measure development. Because mental health problems are difficult to diagnose, claims often contain incorrect information that present challenges to accurate case finding. We attempted to minimize this problem by requiring either an inpatient claim with a primary diagnosis of schizophrenia or two outpatient claims on different days with a primary diagnosis of schizophrenia, adapting definitions used by others (Busch, Frank & Lehman 2004). However, we acknowledge that claims are not an ideal source to identify this population and may provide an undercount of the target population as diagnosis fields are not required for payment of services. Although current guidelines specify follow-up with a mental health provider following hospitalization, performance on our candidate measure is assessed by follow-up with any provider because mental health providers cannot be identified in Medicaid claims.

    Finally, use of MAX data to test the measures limits the external validity of our results. Our MAX analytic study population was purposely limited to Medicaid recipients with claims data so that we could reliably identify patients with schizophrenia and the services they received. As a result, our study population included primarily disabled, non-dual-eligible enrollees in FFS plans. However, this group represents only a minority of the universe of people with SMI who receive mental health treatment through Medicaid programs. In particular, because drugs treatments are reimbursed by Medicare Part D for dually-eligible enrollees we are unable to include them, thus eliminating about 40 percent of all disabled Medicaid recipients from performance assessment.

  2. Several topics were of interest to ASPE, the development team, and stakeholders, but the evidence base, tools, and methods for tracking these measures are immature. For example, evaluating receipt of evidence-based psychosocial services may require measures that address the structures of care (e.g., availability of trained providers, supervision). State officials in particular were interested in measures addressing potential overuse of pharmacological treatments, which is challenging to document in the absence of tools for risk adjustment and symptom measurement. In addition, the evidence to support overuse measures is inconsistent. Patient-reported outcomes were also of interest to stakeholders, but they cannot be ascertained using claims data.

    There was considerable interest in focus groups and TAG on addressing the physical health needs of people with schizophrenia; however, there was not always evidence to provide a rationale for a particular focus on such people for a given test. Some highly important preventive services, in particular tobacco cessation counseling and assistance, are not feasible in claims data. While there was evidence of low rates of cervical cancer screening among women with schizophrenia, there was no such evidence of a gap in care for HIV screening. Continuity of Medicaid enrollment was proposed to assess whether people with schizophrenia have consistent access to services; however, some lapses in coverage may be related to desirable outcomes (such as employment), and it would not be possible to determine the reason for loss of coverage. As the evidence base grows and use of electronic medical records and other electronic data repositories (for example, registries) also grows, so too will the ability to implement evidence-based measures.

  3. Quality measurement for Medicaid recipients with schizophrenia presents implementation issues. During the development process, and in particular during the field-testing process, we became aware of several issues related to measure implementation. Key implementation issues included measure attribution, variations in care financing, and the need for long look-back periods for several measures. For example, although the TAG and several stakeholders endorsed the inclusion of a general measure tracking ED use, some providers voiced concerns about attribution for this measure. Specifically, during the field-testing process, mental health providers felt they should not be held accountable for ED visits for accidents or other non-mental health reasons. Consequently, we dropped the measure of general ED use from our pilot-testing. However, attribution of care processes and outcomes will likely prove controversial, though implementation of the proposed measures at the state (rather than the provider level) will help to minimize concerns over attribution.

    We found that variation in the financing of services for people with SMI limited our ability to measure the care provided by Medicaid programs. For example, the provision of services through state mental health systems, the coverage of mental health services through Medicare for dual-eligible beneficiaries, the prohibition of same-day billing of medical and behavioral health services, and interstate variation in Medicaid and disability standards all underscore the limitations of claims data to measure quality for enrollees with schizophrenia.

    Finally, we found that reliance on Medicaid claims to produce rates of health screening can require a large volume of data to address issues of "look-back" for selected conditions. For example, some health conditions have a screening recommendation of every five years. Therefore, to compute a health screening measure for these conditions, information systems require the capacity to look back over a five-year claims history, which for some states could be a daunting task.

  4. The distinction between enrollees with schizophrenia and other SMI conditions is, in many cases, artificial. The project team, ASPE, and measure stakeholders all expressed the belief that conceptually, many issues related to schizophrenia also apply broadly to people with any SMI. It was outside the scope of this project to conduct the full evidence review and testing necessary for this work. Further work is needed to consider whether measures similar to the ones developed and tested under this contract would be relevant for people with bipolar disorder and other SMI.

View full report


"schqm.pdf" (pdf, 5.77Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®