Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation. A. Opportunities for Psychotherapy Measures


Based on our synthesis of the strengths and limitations of measurement approaches and feedback from our expert panel, we propose prioritizing the measurement of outcomes. This would involve developing the infrastructure for outcomes monitoring and building capacity for delivering evidence-based psychotherapies. Outcome measurement efforts must ensure consumer participation in guiding measure-development, allow flexibility in the choice of measures, protect the privacy and confidentiality of outcomes data, and include incentives and support for providers who make these important but sometimes radical changes in practice. Structure measures may help to facilitate the development of the infrastructure to measure outcomes and ensure that providers have the ability to offer evidence-based treatments. Tracking the use of psychotherapy will be a useful adjunct for these efforts. In the longer term, efforts should focus on further operationalizing and reporting outcome measures and building process measures that will help support quality improvement.

In other areas of health care, process measures have preceded the development and implementation of outcome measures, in part because a strong evidence base and consensus among stakeholders supported the identification of processes expected to lead to positive outcomes (for example, glucose testing for diabetes) and because these processes were relatively straightforward to measure (for example, did the patient receive the glucose test?). Defining these processes of care for psychotherapy is much more complicated and controversial. Although certain types of psychotherapy have strong evidence, there is not widespread agreement in the literature or among clinicians on which specific processes of care or treatment elements lead to positive outcomes. For example, there is not clear evidence that a psychotherapy session that involves assigning homework yields superior outcomes compared with a session that does not (Bell et al. 2013). In addition, the delivery of psychotherapy may be adapted to fit into the context of the service environment and meet the needs of the consumer, but it is unclear to what extent these adaptations in the processes of care influence outcomes. Bickman (2008) summarized that evidence-based treatments in mental health "are not structured in a way that they can be mechanically implemented without variations introduced by the clinician and the service organization. Mental health services will not be successful in removing the influence of the clinician or 'clinician-proofing' treatments any more than the field of education has been successful in 'teacher-proofing' the curriculum." Such adaptations, as well as the lack of research to understand how they influence outcomes, present challenges for developing structure and process measures that can be used for accountability. Focusing on steps that lead to the measurement of outcomes may offer the most value to the broadest group of stakeholders. A focus on outcomes would be consistent with changes underway to measures in medical care, where new guidelines are calling into question some established measures, including those for cardiovascular disease and breast cancer.

As there are limitations with any single measurement approach or data source, it is likely that in the long-term different types of measures that draw on various data sources will be necessary to fully understand how psychotherapy is delivered and whether it yields positive outcomes. Using multiple measures could generate information that helps to identify the relationship between structure, process, and outcomes in psychotherapy. For example, information from structure measures, such as whether providers have certain types of training and use specific procedures, could be coupled with data from EHRs and outcome assessments to elucidate the link between the structures that support psychotherapy, the delivery of specific psychotherapeutic content, and the outcomes of treatment. Measuring outcomes while simultaneously trying to understand the structures and processes of care may help to identify the most effective forms of psychotherapy and build the evidence for those psychotherapies with less empirical support (Garland et al. 2010a).

1. Immediate Opportunities

Implement structure measures that build capacity for delivering evidence-based psychotherapies and lead to the measurement of outcomes. As described above, there are several examples of practical tools and systems that have been used to assess the outcomes of psychotherapy. Several of these tools and measurement systems have a substantial literature supporting their effectiveness in improving outcomes and use in large-scale quality improvement initiatives; in some ways, their use has become an evidence-based practice. Some of these tools and systems have been used primarily for psychotherapy research or clinical training, and others have been implemented by large provider networks and health plans. The basic infrastructure to support the measurement of psychotherapy outcomes--including such things as patient registries and electronic data collection and feedback systems--is available to some providers via health plans or delivery systems and could serve as a model. Nonetheless, the widespread adoption of routine outcomes monitoring for psychotherapy would require further investments in the data collection and reporting infrastructure.

Structure measures that assess the capacity to deliver psychotherapy could draw upon existing methods for assessing fidelity in research settings and standards for other aspects of evidence-based practices that are incorporated in requirements for health homes. These measures could address training and supervision of staff as well as whether providers/entities have protocols and systems in place to conduct routine consumer assessments, match treatment to consumer needs and preferences, and adapt treatment when consumers do not improve on outcome measures.

States could position the structure measures within the certification requirements of their Medicaid and mental health agencies, health plans, and ACOs for provider networks. These structure measures could also be used within health homes, patient-centered medical homes, or other certification programs. These measures would ideally focus on mental health clinics, primary care practices, or other provider organizations rather than individual clinicians or therapists. They would focus on whether the organization has staff with basic training and credentials in evidence-based psychotherapy, protocols for supervision and stepped care, and the infrastructure to measure outcomes and use outcomes data to inform clinical decision making and quality improvement. States and health plans already have some basic structure measures in place to determine whether individual clinicians meet licensing requirements, and some states impose additional requirements for mental health providers to bill for specific types of services. States that require formal credentialing in specific evidence-based treatments and health plans that offer outcomes monitoring infrastructure may serve as examples for future efforts.

Offer outcome measurement approaches that are flexible but allow for comparisons. Many tools could be used to measure outcomes; each has strengths and limitations, and applications to various populations and settings. Health plans, clinics, and providers may benefit from selecting among tools that are applicable to the population and community they serve and that can be used in the context of their particular service setting. As mentioned above, there are some repositories of tools that could be useful. Consistent with the principles of other patient-reported outcome measures, the tool selected must be reliable and valid; "person centered"; meaningful for consumers, their families, and providers; and amenable to change (NQF 2013). Moreover, they must be easy to administer and interpret, they must yield information that providers and consumers can use to engage in the discussion of treatment goals, and they must foster progress toward reaching those goals. Finally, the systems that support or collect these measures must offer different modes to collecting the data (web-based, mail, telephone) and present the data back to providers and consumers in a manner that is useful for making treatment decisions and informing quality improvement. Some of the examples described in this paper may serve as models.

Offer incentives, training, and ongoing support. We should not underestimate the extent to which the routine measurement of outcomes and the use of outcomes data to guide clinical decision making and monitor quality introduce a major paradigm shift for many providers. Some providers are not accustomed to using standardized tools or assessments to monitor progress, and they may feel uncomfortable reporting information to other entities. It would be naïve to think that providers will change practice only because there is literature supporting the benefits of routine outcomes monitoring (Boswell et al. 2013).

Although payers and health plans have a financial incentive to ensure that providers monitor and obtain positive outcomes in an effort to minimize the use of more costly services, in the current reimbursement environment, measuring outcomes (or collecting process measures) would impose burden on providers--particularly small or solo practices that are unaffiliated with larger health care delivery systems. Many of these providers do not have the staff or other resources to complete and submit routine measures. Within the flexibility of Medicaid health homes and managed care arrangements, states and health plans could offer incentives for collecting and reporting psychotherapy outcomes and for achieving desirable outcomes (Bao et al. 2013). These incentives could come in the form of actual payment for the administration and reporting of outcome measures or by increasing referrals to high-performing providers (Boswell et al. 2013). Providers could also receive incentives for adopting some of the structures that may support outcomes monitoring, such as adoption of clinical registries and systems for collecting data or by using existing systems to demonstrate significant patient improvement.

Moreover, providers who are accustomed to relying solely on their clinical judgment will need training and support to understand how to integrate the results of routine measures into their decisions about how to tailor individual treatments and how to improve the effectiveness of the care for all consumers (Boswell et al. 2013). Such training could take the form of continuing education programs or training in graduate programs. Following the examples described in this paper, health plans may be well-positioned to offer decision support by giving feedback to providers in a manner that offers guidance on best practices and gives providers an opportunity to reflect on difficult cases.

Ensure confidentiality, security, and appropriate use of data. Providers, consumers, and their families need assurances that the sensitive information they provide is confidential and secure and will be used for the specific purposes. State agencies, health plans, or other entities must also be transparent in what information will be publically reported.

Individual providers may be reluctant to have their performance publically reported or they may have an insufficient number of consumers to report on. Attributing outcomes to an individual provider is also questionable, given that individuals often receive care from several providers and the very real possibility that factors outside of the provider's control account for outcomes. The challenge of attributing outcomes to a provider might argue for measuring outcomes at the patient level for clinical decision making but publicly reporting only at the health plan or clinic/organization level, which would align with other efforts to measure system performance (Berenson et al. 2013; Conway et al. 2013). Such an approach could also encourage a more collaborative effort focused on quality improvement at the organizational level.

Engage consumers and other stakeholders in the development of outcome measurement systems. As state agencies, health plans, and other entities engage in developing and implementing measurement strategies, they must ensure that the measures are consistent with the goals, values, and cultural diversity of consumers and their families, as well as other stakeholders. Consumers and family members must be engaged in selecting salient measures and developing processes that provide usable feedback from those measures. In clinical practice, consumers and their family members will benefit from having a choice of measures that cover domains of functioning that represent meaningful progress toward reaching goals rather than measures that only assess the remediation of symptoms or satisfaction with care. Likewise, payers, health plans, state and county agencies, and providers must provide input into the selection of measures.

The outcomes of psychotherapy are also relevant to stakeholder groups that may be somewhat outside of mainstream health or behavioral health care, such as education and juvenile justice systems or other state agencies. It is important to consider how these measures could be informative for these other groups. For instance, education systems may be interested in which provider organizations achieve outcomes that are meaningful for school systems, such as absenteeism.

Track the use of psychotherapy. Given that many individuals with mental health problems do not receive psychotherapy (Brown et al. 2012), there may be value in using claims data to track the use of services. As described above, currently available claims data, and Medicaid data in particular, lack information on the content of the therapy provided, and there are limitations in using this data to make comparisons across providers, health plans, or states. Nonetheless, claims data could be used to understand whether individuals with an indicated condition or medication have any contact with the mental health care system and maintain that contact. Such measures may not yield information that is necessarily indicative of the quality of psychotherapy, but they could point to patterns of use that merit further investigation.

2. Longer-Term Opportunities

In this section we describe some of the activities that could facilitate refining and reporting outcome measures and improve the data sources that would facilitate process measures that support quality improvement.

Clarify the conceptualization of outcomes and demonstrate methods for reporting outcomes measures. Although there are many examples of outcome measurement approaches, work remains to further refine the methods for using outcome measurement strategies for accountability. Measures that are primarily used for clinical supervision or training may not be appropriate for the purposes of accountability or large-scale quality improvement. As described in the examples above, the outcomes in many of these measurement systems are conceptualized quite differently. Some focus on remission of symptoms and functioning within specific domains of mental health, such as depression, but others are much broader. As measurement systems are implemented and data accumulates, it will be important to understand how different conceptualizations and strategies for measuring outcomes function among different populations and in different contexts. With more data, systems may be more equipped to implement measures that are applicable to broad diagnostic and demographic groups and can be used across settings.

Given the many different ways in which outcomes could be conceptualized and measured, there may be opportunities and particular value in building on ongoing efforts to design measures around goal attainment, which is consumer-directed. More research is needed to understand how goal attainment scales should be calculated for the purposes of accountability. Further, there is a need for measurement experts to refine risk adjustment strategies and look for alternatives to risk adjustment to hold entities accountable for care and facilitate fair comparisons.

Enhance billing codes to facilitate the use of claims data for quality measurement. In the future, measures that capture the content and duration of psychotherapy could theoretically use claims data if there were major improvements in the specificity of billing codes and if those codes were tied to some type of credentialing or linked to outcomes. CMS, state Medicaid programs, and health plans could develop more descriptive billing codes that correspond to specific psychotherapeutic processes, and they could restrict the use of such codes to providers who demonstrate competency or have certain credentials (as is the case with some other mental health billing codes, such as Multisystemtic Therapy or Assertive Community Treatment). Again, these processes are currently not well-defined for psychotherapy, and it would take considerable effort to achieve consensus among professional societies, payers, health plans, and state agencies on which processes of care are linked to outcomes and should therefore receive their own billing codes, and possibly differential payment. In addition, given that providers typically belong to many different health plans and receive reimbursement through various streams, it is a considerable investment for state Medicaid programs or health plans to create new codes and encourage their use. The use of new coding schemes would also impose an administrative and time burden on providers, especially if those schemes varied across health plans or other payers. Even with some standardization of codes, it is likely that their use will always vary somewhat across providers, states, and health plans, given the differences in how providers are reimbursed at the state and local levels. This might limit comparisons.

Incorporate fields on the content of psychotherapy in EHRs. EHRs could incorporate data elements that capture delivery of specific psychotherapeutic content. The VA is beginning to capture some specific elements of psychotherapy in its EHRs, which may yield findings about the feasibility of this measurement approach (Kilbourne et al. 2010). Measures that require documentation from EHRs would not currently capture information from the majority of behavioral health providers because most of them have not adopted EHRs, especially providers in small or solo practices who are not affiliated with a larger health care system. Measures that rely on EHRs to capture specific treatment elements would require providers to change their documentation practices--potentially creating additional documentation burden while becoming a "checkbox" measure. An alternative approach to having standardized fields in EHRs would be to develop more sophisticated computerized methods for extracting psychotherapeutic content from the notes of medical records. This approach has been tested and has produced some promising results (Shiner et al. 2013). However, much more work is needed for it to be applied to measures used for quality improvement and accountability. Moreover, the current reporting programs that use information from EHRs rely on structured fields.

Incorporate reporting on the content of psychotherapy into consumer surveys. Measures that examine the content of psychotherapy could use data reported directly from consumers. Health plans routinely conduct surveys to assess consumers' experiences with care, and they could also attempt to have consumers report on whether they received specific psychotherapeutic content. Although several widely used surveys assess consumers' experience with mental health treatment, they assess care in a more global fashion and do not attempt to measure whether the individual received specific psychotherapeutic content (Eisen et al. 2001). The work of Miranda and colleagues (2010) is a promising first step toward developing measures that attempt to elicit the psychotherapeutic content of visits directly from the consumer. More extensive testing of these measures would be needed to examine not only the extent to which consumers can accurately recall the content of psychotherapy but also how the reports are linked to outcomes.

Expand research on the effective ingredients of psychotherapy, with particular focus on specific consumer populations and conditions. Several gaps in research impede the development of quality measures for psychotherapy. There is a particular need to further identify the "active" ingredients as well as the common factors and therapeutic treatment processes that are associated with improvements in outcomes across different types of psychotherapy. Such research would help to inform the development of process measures that could be used across multiple types of therapy. As noted above, there is a need to better understand the effectiveness of various forms of psychotherapy among consumers with varying mental health needs and cultural backgrounds.

New measures would require testing to understand the best scoring method. In particular, there are no clear guidelines or standards for scoring the structure and process measures discussed in this paper. Some of the measures could be scored as an all-or-nothing measure. Alternatively, measures that assess whether providers deliver therapy as intended could be scored using some minimal threshold, as has been done in some studies (Barber et al. 2007). As researchers have noted, the thresholds vary across studies, and little information exists on the rationale behind the selection of the threshold. When developing a scoring algorithm, it is important to consider how the score will be used and the potential implications of high or low scores. For measures that use information from medical records/EHRs or surveys, the extent to which the delivery of specific content in a single therapy session is associated with consumer outcomes is unclear. It is conceivable that a provider could have one bad session and deliver high quality care in all other sessions so that, taken together, their sessions result in positive outcomes. In high-stakes situations where the score has significant implications, it might be important to consider using an overall score that presents a fair and accurate assessment of the quality of care provided. It might also make sense to score some types of measures only for an organization or health plan, not for individual providers.


View full report


"QualPsy.pdf" (pdf, 445.96Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®