Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation

People with depression, anxiety, post-traumatic stress disorder, and other mental disorders can benefit from evidence-based psychotherapy. However, the extent to which these therapies are available and implemented effectively is unclear as few performance measures of psychotherapy care exist. Of the psychotherapy measures identified none are endorsed by the National Quality Forum. Although Medicaid is the single largest payer of mental health services in the United States, a recent search of the U.S. Department of Health and Human Services (HHS), Centers for Medicare and Medicaid Services (CMS) Measures Inventory ("measures used by CMS in various quality, reporting and payment programs") identified no "psychotherapy" measures currently in use by CMS. Of the psychotherapy measures that do exist, the vast majority focus on access to care or quantifying the number of visits rather than on the content of care (e.g., if protocol for care was followed) or outcomes of care.

The Affordable Care Act encourages health care delivery system reforms that use budgeted payment systems coupled with performance measures to both disincentivize inefficient use of health care resources while also ensuring accountability for providing access to quality care. There is an increased need for performance measures to monitor the delivery and outcomes of psychotherapies in this context. Such measures would inform quality improvement and allow stakeholders to assess whether the services purchased were delivered as intended and achieved desired outcomes. These measures would allow payers to reimburse for the actual delivery (or non-delivery) of evidence-based psychotherapy.

To advance the development of such measures, the HHS Office of the Assistant Secretary for Planning and Evaluation in partnership with the HHS Substance Abuse and Mental Health Services Administration contracted with Mathematica Policy Research and the National Committee for Quality Assurance to examine the strengths and limitations of different measurement strategies that could be used to assess the delivery of psychotherapy for the purposes of quality improvement and accountability.

In this paper, we describe how structure, process, and outcome measures could be used to monitor and improve the delivery of psychotherapy. Such measures could be used to inform health care delivery system reforms that are influencing coverage and payment policies and quality of care going forward. We review the strengths and limitations of each type of measure and the data sources that could be used to support them. We focus on measures assessing the effectiveness and outcomes of care rather than other domains, such as measures that assess the utilization or costs of services. Table ES-1 is a summary of the measurement options.

Structure measures would assess the capacity of a provider, clinic, or health care organization to deliver evidence-based psychotherapy and monitor the outcomes of care. Such measures would document whether policies and procedures are implemented to support psychotherapy, to assess provider training and competency, and to support collection of consumer or clinician reported data on symptoms and functioning. Although such measures would require new investments for providers and health systems, they could be an important step in improving the delivery of evidence-based care and outcomes. Structure measures could be incorporated into existing efforts to test new models of delivering care.

Process measures would assess whether individuals receive psychotherapy that resembles evidence-based care. Claims data could be used to measure the frequency, duration, or continuity of psychotherapy visits, but the data lack detail on the psychotherapeutic content of visits. Measures based on these data could incentivize more visits of unknown quality. Process measures that use data from medical records/electronic health records (EHRs) could examine the delivery of specific psychotherapeutic content; web-based systems for delivering therapy could also document care received. Measures developed from these data sources would require major changes in how providers document care and may change the way consumers receive care. These measures have the potential to become superficial "checkbox" measures. Finally, providers and consumers could report the content of psychotherapy through surveys. Although there are promising findings from pilot studies of surveys that have attempted to gather such information, such surveys would be costly to implement and subject to response bias and their viability as stand-alone measures for accountability is unclear.

Outcome measures would assess whether individuals receiving psychotherapy experience improvement in mental health symptoms and functioning across various domains of life (for example, employment, family life, social functioning). Measuring outcomes could involve using repeated standardized assessments that yield information providers and consumers can use to inform treatment decisions, while in the aggregate, the data can be used for quality monitoring and improvement. There is strong evidence that such an approach--known as measurement-based care or routine outcomes monitoring--helps identify consumers at risk for treatment failure and improves outcomes. Several health plans and large quality improvement initiatives have used this approach to give feedback to providers and consumers and inform quality improvement. Outcome measures are attractive because they assess what is important to consumers and their families and overcome some of the limitations of process and structural measures--namely that they focus on whether an individual is actually improving or whether a provider's overall population of clients is improving to a degree that could be reasonably expected. What is lacking, however, is the widespread infrastructure to support the collection of outcomes data as well as agreement on the methods for reporting these results in ways that will allow for fair comparisons. Further, such an approach would introduce a major change in the way that some providers deliver care and consumers participate in care.

Given the complexity of measuring psychotherapy and the limitations of any single measurement approach, using different types of measures would provide a comprehensive understanding of the quality of psychotherapy and insight into the link between the structures, processes, and outcomes in psychotherapy. In order to advance quality measurement in psychotherapy, we propose that policymakers, state agencies, health plans, providers, consumers, measure developers, and the larger mental health community engage n the following activities:

• Prioritize the measurement of outcomes. In other areas of health care, process measures have preceded outcome measures, in part, because the processes of care were well-defined and their links to outcomes were based on strong evidence. Although there is strong evidence to support various forms of psychotherapy, there is neither widespread agreement on the specific processes or components of psychotherapy that lead to improvements in outcomes nor consensus on how to best measure them. This remains a subject of ongoing research and debate. There is, however, agreement that psychotherapy should improve outcomes that are meaningful to consumers and their families. There may be an opportunity for psychotherapy outcome measures to advance without waiting on process measures. As described in this report, there are promising examples of outcome measurement systems in psychotherapy. These systems rely on repeated assessments of symptoms and functioning, which providers can use to tailor treatment and consumers can use for self-monitoring and to inform treatment decisions. In the aggregate, though there remain technical concerns that limit these measures' utility for accountability purposes, the data from these systems can be used for quality improvement. For example, an organization could be incentivized to meet minimal thresholds for expected consumer clinical improvement. Although investments would be needed to expand the infrastructure to support the implementation of outcome measures, there are many examples and resources on which to build.

• Develop structure measures that promote the use of best practices and the measurement of outcomes. The credentialing process for health plan and provider networks currently focus on licensure; some states are going further by having providers demonstrate their competencies in order to receive reimbursement for specific psychotherapy treatments. There are also examples of health plans and community initiatives focused on measuring symptom reduction and functional outcomes. Expanding on these efforts, state and health plans could use structure measures to require providers and clinics to demonstrate use of evidence-based care. Structure measures could assess whether providers have systematic processes for training and supervision of staff, data systems for monitoring symptoms and functioning, and protocols for delivering evidence-based treatment and adjusting treatment based on outcome measures.

• Track use of psychotherapy. There are several limitations to using claims data to measure the quality of psychotherapy, but these data could be useful for tracking service use. Health plans and others could use claims to identify individuals with specific diagnoses or those receiving medications to examine their contact with the mental health care system. This type of a rudimentary measure would not assess the quality of care, but could provide insight into service utilization patterns that merit further investigation.

• Continue to refine measures and data systems for reporting. As measures are implemented, there will need to be ongoing efforts to maintain and refine them. For outcome measures, such refinement could address the selection of tools, challenges from statistical artifacts such as regression to the mean, risk adjustment, and other ways to account for differences in patient populations. Evidence-based processes of care could get incorporated into EHRs and claims; for example, EHRs could incorporate fields for reporting the delivery of specific treatment elements, and state Medicaid programs or health plans could use new billing codes to reimburse specific processes of care with strong evidence. This would require considerable investments in the data collection infrastructure, consensus-based efforts by multiple stakeholders to standardize billing codes and EHR fields, and efforts to support widespread adoption--all of which may be longer-term endeavors that rely on accumulating stronger evidence that specific processes of care are associated with outcomes. Structure and process measures would not replace outcome measures, but they may offer guidance on best practices and aid in quality improvement.

Quality measures are one tool that can improve psychotherapy outcomes. To have the greatest impact, a strategy for developing and implementing such measures should account for the strengths and limitations of different types of measures as well as opportunities for aligning quality improvement efforts and delivery system reforms. Support from a wide range of stakeholders will be needed to advance the implementation of measures that seek to improve the quality and outcomes of psychotherapy.