Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation

Publication Date
Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation
Executive Summary
May 15, 2014
Jonathan Brown, Sarah Hudson Scholle and Melissa Azur
Mathematica Policy Research
Abstract
Various types of psychotherapy can benefit people who experience depression, anxiety, and other common mental disorders. However, the extent to which these psychotherapies are available and delivered effectively is largely unknown. In the context of health care reforms that encourage performance measurement, a strong set of measures focused on psychotherapy could help stakeholders understand what psychotherapy services are delivered and if they yield positive outcomes. This paper discusses the advantages and challenges of structure, process, and outcome measures for psychotherapy. It concludes with several recommendations for advancing the measurement of psychotherapy. DISCLAIMER: The opinions and views expressed in this report are those of the authors. They do not necessarily reflect the views of the Department of Health and Human Services, the contractor or any other funding organization.
This Executive Summary is available on the Internet at:

People with depression, anxiety, post-traumatic stress disorder, and other mental disorders can benefit from evidence-based psychotherapy. However, the extent to which these therapies are available and implemented effectively is unclear as few performance measures of psychotherapy care exist. Of the psychotherapy measures identified none are endorsed by the National Quality Forum. Although Medicaid is the single largest payer of mental health services in the United States, a recent search of the U.S. Department of Health and Human Services (HHS), Centers for Medicare and Medicaid Services (CMS) Measures Inventory ("measures used by CMS in various quality, reporting and payment programs") identified no "psychotherapy" measures currently in use by CMS. Of the psychotherapy measures that do exist, the vast majority focus on access to care or quantifying the number of visits rather than on the content of care (e.g., if protocol for care was followed) or outcomes of care.

The Affordable Care Act encourages health care delivery system reforms that use budgeted payment systems coupled with performance measures to both disincentivize inefficient use of health care resources while also ensuring accountability for providing access to quality care. There is an increased need for performance measures to monitor the delivery and outcomes of psychotherapies in this context. Such measures would inform quality improvement and allow stakeholders to assess whether the services purchased were delivered as intended and achieved desired outcomes. These measures would allow payers to reimburse for the actual delivery (or non-delivery) of evidence-based psychotherapy.

To advance the development of such measures, the HHS Office of the Assistant Secretary for Planning and Evaluation in partnership with the HHS Substance Abuse and Mental Health Services Administration contracted with Mathematica Policy Research and the National Committee for Quality Assurance to examine the strengths and limitations of different measurement strategies that could be used to assess the delivery of psychotherapy for the purposes of quality improvement and accountability.

In this paper, we describe how structure, process, and outcome measures could be used to monitor and improve the delivery of psychotherapy. Such measures could be used to inform health care delivery system reforms that are influencing coverage and payment policies and quality of care going forward. We review the strengths and limitations of each type of measure and the data sources that could be used to support them. We focus on measures assessing the effectiveness and outcomes of care rather than other domains, such as measures that assess the utilization or costs of services. Table ES-1 is a summary of the measurement options.

TABLE ES-1. Strategies for Measuring the Quality of Psychotherapy
  Measure  
Type
Goal   Potential Data  
Source(s)
Strengths Challenges
Structure   Assess the capacity of providers to offer evidence-based psychotherapy Documentation and on-site audits Provides guidance on evidence-based practices; could be tied to payment and credentialing Measures could be expensive and burdensome
Process Assess whether psychotherapy is delivered with fidelity to evidence-based model Claims Medical records/EHRs Provider or consumer surveys Claims and EHRs are low-cost/low-burden Surveys could assess what consumers receive from their own perspective (e.g., CAHPS® like surveys) Claims lack detail on content of visits EHRs not currently widely used in mental health care and would need to contain necessary data elements Surveys are expensive and validity of reports for measuring content of psychotherapy is unknown
Outcome Assess improvements in outcomes among individuals who receive psychotherapy Standardized measures of symptom severity and functioning Measures what matters to consumers, families, and other stakeholders Lack of widespread infrastructure to support measures

Structure measures would assess the capacity of a provider, clinic, or health care organization to deliver evidence-based psychotherapy and monitor the outcomes of care. Such measures would document whether policies and procedures are implemented to support psychotherapy, to assess provider training and competency, and to support collection of consumer or clinician reported data on symptoms and functioning. Although such measures would require new investments for providers and health systems, they could be an important step in improving the delivery of evidence-based care and outcomes. Structure measures could be incorporated into existing efforts to test new models of delivering care.

Process measures would assess whether individuals receive psychotherapy that resembles evidence-based care. Claims data could be used to measure the frequency, duration, or continuity of psychotherapy visits, but the data lack detail on the psychotherapeutic content of visits. Measures based on these data could incentivize more visits of unknown quality. Process measures that use data from medical records/electronic health records (EHRs) could examine the delivery of specific psychotherapeutic content; web-based systems for delivering therapy could also document care received. Measures developed from these data sources would require major changes in how providers document care and may change the way consumers receive care. These measures have the potential to become superficial "checkbox" measures. Finally, providers and consumers could report the content of psychotherapy through surveys. Although there are promising findings from pilot studies of surveys that have attempted to gather such information, such surveys would be costly to implement and subject to response bias and their viability as stand-alone measures for accountability is unclear.

Outcome measures would assess whether individuals receiving psychotherapy experience improvement in mental health symptoms and functioning across various domains of life (for example, employment, family life, social functioning). Measuring outcomes could involve using repeated standardized assessments that yield information providers and consumers can use to inform treatment decisions, while in the aggregate, the data can be used for quality monitoring and improvement. There is strong evidence that such an approach--known as measurement-based care or routine outcomes monitoring--helps identify consumers at risk for treatment failure and improves outcomes. Several health plans and large quality improvement initiatives have used this approach to give feedback to providers and consumers and inform quality improvement. Outcome measures are attractive because they assess what is important to consumers and their families and overcome some of the limitations of process and structural measures--namely that they focus on whether an individual is actually improving or whether a provider's overall population of clients is improving to a degree that could be reasonably expected. What is lacking, however, is the widespread infrastructure to support the collection of outcomes data as well as agreement on the methods for reporting these results in ways that will allow for fair comparisons. Further, such an approach would introduce a major change in the way that some providers deliver care and consumers participate in care.

Given the complexity of measuring psychotherapy and the limitations of any single measurement approach, using different types of measures would provide a comprehensive understanding of the quality of psychotherapy and insight into the link between the structures, processes, and outcomes in psychotherapy. In order to advance quality measurement in psychotherapy, we propose that policymakers, state agencies, health plans, providers, consumers, measure developers, and the larger mental health community engage n the following activities:

  • Prioritize the measurement of outcomes. In other areas of health care, process measures have preceded outcome measures, in part, because the processes of care were well-defined and their links to outcomes were based on strong evidence. Although there is strong evidence to support various forms of psychotherapy, there is neither widespread agreement on the specific processes or components of psychotherapy that lead to improvements in outcomes nor consensus on how to best measure them. This remains a subject of ongoing research and debate. There is, however, agreement that psychotherapy should improve outcomes that are meaningful to consumers and their families. There may be an opportunity for psychotherapy outcome measures to advance without waiting on process measures. As described in this report, there are promising examples of outcome measurement systems in psychotherapy. These systems rely on repeated assessments of symptoms and functioning, which providers can use to tailor treatment and consumers can use for self-monitoring and to inform treatment decisions. In the aggregate, though there remain technical concerns that limit these measures' utility for accountability purposes, the data from these systems can be used for quality improvement. For example, an organization could be incentivized to meet minimal thresholds for expected consumer clinical improvement. Although investments would be needed to expand the infrastructure to support the implementation of outcome measures, there are many examples and resources on which to build.

  • Develop structure measures that promote the use of best practices and the measurement of outcomes. The credentialing process for health plan and provider networks currently focus on licensure; some states are going further by having providers demonstrate their competencies in order to receive reimbursement for specific psychotherapy treatments. There are also examples of health plans and community initiatives focused on measuring symptom reduction and functional outcomes. Expanding on these efforts, state and health plans could use structure measures to require providers and clinics to demonstrate use of evidence-based care. Structure measures could assess whether providers have systematic processes for training and supervision of staff, data systems for monitoring symptoms and functioning, and protocols for delivering evidence-based treatment and adjusting treatment based on outcome measures.

  • Track use of psychotherapy. There are several limitations to using claims data to measure the quality of psychotherapy, but these data could be useful for tracking service use. Health plans and others could use claims to identify individuals with specific diagnoses or those receiving medications to examine their contact with the mental health care system. This type of a rudimentary measure would not assess the quality of care, but could provide insight into service utilization patterns that merit further investigation.

  • Continue to refine measures and data systems for reporting. As measures are implemented, there will need to be ongoing efforts to maintain and refine them. For outcome measures, such refinement could address the selection of tools, challenges from statistical artifacts such as regression to the mean, risk adjustment, and other ways to account for differences in patient populations. Evidence-based processes of care could get incorporated into EHRs and claims; for example, EHRs could incorporate fields for reporting the delivery of specific treatment elements, and state Medicaid programs or health plans could use new billing codes to reimburse specific processes of care with strong evidence. This would require considerable investments in the data collection infrastructure, consensus-based efforts by multiple stakeholders to standardize billing codes and EHR fields, and efforts to support widespread adoption--all of which may be longer-term endeavors that rely on accumulating stronger evidence that specific processes of care are associated with outcomes. Structure and process measures would not replace outcome measures, but they may offer guidance on best practices and aid in quality improvement.

Quality measures are one tool that can improve psychotherapy outcomes. To have the greatest impact, a strategy for developing and implementing such measures should account for the strengths and limitations of different types of measures as well as opportunities for aligning quality improvement efforts and delivery system reforms. Support from a wide range of stakeholders will be needed to advance the implementation of measures that seek to improve the quality and outcomes of psychotherapy.


The Full Report is also available from the DALTCP website (http://aspe.hhs.gov/office_specific/daltcp.cfm) or directly at http://aspe.hhs.gov/daltcp/reports/2014/QualPsy.cfm.
Product Type
White Paper