Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation

DISCLAIMER: The opinions and views expressed in this report are those of the authors. They do not necessarily reflect the views of the Department of Health and Human Services, the contractor or any other funding organization.

Acknowledgments

Mathematica Policy Research and the National Committee for Quality Assurance (NCQA) prepared this report under contract to the Office of the Assistant Secretary for Planning and Evaluation (ASPE), U.S. Department of Health and Human Services (HHS) (HHSP23320100019WI/ HHSP23337001T). Additional funding was provided by the HHS Substance Abuse and Mental Health Services Administration (SAMHSA). The authors appreciate the guidance of Kirsten Beronio and Joel Dubenitz (ASPE), D.E.B. Potter (HHS Agency for Healthcare Research and Quality [AHRQ]), and Lisa Patton (SAMHSA). The report incorporates feedback from Henry Ireys (Mathematica), Richard Frank (Harvard University), Harold Pincus (Columbia University) and a technical expert panel that included consumers, providers, performance measurement experts, and representatives from health plans and state agencies. Panelists included Francisca Azocar (Optum Behavioral Solutions), James Boswell (State University of New York), Lynn Bufka (American Psychological Association), Mirean Coleman (National Association of Social Workers), Eric Hamilton (ValueOptions), Kimberly Hoagwood (New York University), Jeanne Miranda (University of California, Los Angeles), Keris Myrick (National Alliance for the Mentally Ill and Project Return Peer Support Network), and Nathan Tatro (American Psychological Association).

The views and opinions expressed in this report are those of the authors and do not necessarily reflect the views, opinions, or policies of ASPE, SAMHSA, AHRQ, or HHS, or the technical expert panel. The measures described in this report are provided as examples; their inclusion should not be interpreted as an endorsement by HHS, ASPE, SAMHSA, AHRQ, NCQA, or Mathematica Policy Research. The authors are solely responsible for any errors.

Acronyms

The following acronyms are mentioned in this report and/or appendix.

Executive Summary

People with depression, anxiety, post-traumatic stress disorder, and other mental disorders can benefit from evidence-based psychotherapy. However, the extent to which these therapies are available and implemented effectively is unclear as few performance measures of psychotherapy care exist. Of the psychotherapy measures identified none are endorsed by the National Quality Forum. Although Medicaid is the single largest payer of mental health services in the United States, a recent search of the U.S. Department of Health and Human Services (HHS), Centers for Medicare and Medicaid Services (CMS) Measures Inventory ("measures used by CMS in various quality, reporting and payment programs") identified no "psychotherapy" measures currently in use by CMS. Of the psychotherapy measures that do exist, the vast majority focus on access to care or quantifying the number of visits rather than on the content of care (e.g., if protocol for care was followed) or outcomes of care.

The Affordable Care Act encourages health care delivery system reforms that use budgeted payment systems coupled with performance measures to both disincentivize inefficient use of health care resources while also ensuring accountability for providing access to quality care. There is an increased need for performance measures to monitor the delivery and outcomes of psychotherapies in this context. Such measures would inform quality improvement and allow stakeholders to assess whether the services purchased were delivered as intended and achieved desired outcomes. These measures would allow payers to reimburse for the actual delivery (or non-delivery) of evidence-based psychotherapy.

To advance the development of such measures, the HHS Office of the Assistant Secretary for Planning and Evaluation in partnership with the HHS Substance Abuse and Mental Health Services Administration contracted with Mathematica Policy Research and the National Committee for Quality Assurance to examine the strengths and limitations of different measurement strategies that could be used to assess the delivery of psychotherapy for the purposes of quality improvement and accountability.

In this paper, we describe how structure, process, and outcome measures could be used to monitor and improve the delivery of psychotherapy. Such measures could be used to inform health care delivery system reforms that are influencing coverage and payment policies and quality of care going forward. We review the strengths and limitations of each type of measure and the data sources that could be used to support them. We focus on measures assessing the effectiveness and outcomes of care rather than other domains, such as measures that assess the utilization or costs of services. Table ES-1 is a summary of the measurement options.

Structure measures would assess the capacity of a provider, clinic, or health care organization to deliver evidence-based psychotherapy and monitor the outcomes of care. Such measures would document whether policies and procedures are implemented to support psychotherapy, to assess provider training and competency, and to support collection of consumer or clinician reported data on symptoms and functioning. Although such measures would require new investments for providers and health systems, they could be an important step in improving the delivery of evidence-based care and outcomes. Structure measures could be incorporated into existing efforts to test new models of delivering care.

Process measures would assess whether individuals receive psychotherapy that resembles evidence-based care. Claims data could be used to measure the frequency, duration, or continuity of psychotherapy visits, but the data lack detail on the psychotherapeutic content of visits. Measures based on these data could incentivize more visits of unknown quality. Process measures that use data from medical records/electronic health records (EHRs) could examine the delivery of specific psychotherapeutic content; web-based systems for delivering therapy could also document care received. Measures developed from these data sources would require major changes in how providers document care and may change the way consumers receive care. These measures have the potential to become superficial "checkbox" measures. Finally, providers and consumers could report the content of psychotherapy through surveys. Although there are promising findings from pilot studies of surveys that have attempted to gather such information, such surveys would be costly to implement and subject to response bias and their viability as stand-alone measures for accountability is unclear.

Outcome measures would assess whether individuals receiving psychotherapy experience improvement in mental health symptoms and functioning across various domains of life (for example, employment, family life, social functioning). Measuring outcomes could involve using repeated standardized assessments that yield information providers and consumers can use to inform treatment decisions, while in the aggregate, the data can be used for quality monitoring and improvement. There is strong evidence that such an approach--known as measurement-based care or routine outcomes monitoring--helps identify consumers at risk for treatment failure and improves outcomes. Several health plans and large quality improvement initiatives have used this approach to give feedback to providers and consumers and inform quality improvement. Outcome measures are attractive because they assess what is important to consumers and their families and overcome some of the limitations of process and structural measures--namely that they focus on whether an individual is actually improving or whether a provider's overall population of clients is improving to a degree that could be reasonably expected. What is lacking, however, is the widespread infrastructure to support the collection of outcomes data as well as agreement on the methods for reporting these results in ways that will allow for fair comparisons. Further, such an approach would introduce a major change in the way that some providers deliver care and consumers participate in care.

Given the complexity of measuring psychotherapy and the limitations of any single measurement approach, using different types of measures would provide a comprehensive understanding of the quality of psychotherapy and insight into the link between the structures, processes, and outcomes in psychotherapy. In order to advance quality measurement in psychotherapy, we propose that policymakers, state agencies, health plans, providers, consumers, measure developers, and the larger mental health community engage n the following activities:

• Prioritize the measurement of outcomes. In other areas of health care, process measures have preceded outcome measures, in part, because the processes of care were well-defined and their links to outcomes were based on strong evidence. Although there is strong evidence to support various forms of psychotherapy, there is neither widespread agreement on the specific processes or components of psychotherapy that lead to improvements in outcomes nor consensus on how to best measure them. This remains a subject of ongoing research and debate. There is, however, agreement that psychotherapy should improve outcomes that are meaningful to consumers and their families. There may be an opportunity for psychotherapy outcome measures to advance without waiting on process measures. As described in this report, there are promising examples of outcome measurement systems in psychotherapy. These systems rely on repeated assessments of symptoms and functioning, which providers can use to tailor treatment and consumers can use for self-monitoring and to inform treatment decisions. In the aggregate, though there remain technical concerns that limit these measures' utility for accountability purposes, the data from these systems can be used for quality improvement. For example, an organization could be incentivized to meet minimal thresholds for expected consumer clinical improvement. Although investments would be needed to expand the infrastructure to support the implementation of outcome measures, there are many examples and resources on which to build.

• Develop structure measures that promote the use of best practices and the measurement of outcomes. The credentialing process for health plan and provider networks currently focus on licensure; some states are going further by having providers demonstrate their competencies in order to receive reimbursement for specific psychotherapy treatments. There are also examples of health plans and community initiatives focused on measuring symptom reduction and functional outcomes. Expanding on these efforts, state and health plans could use structure measures to require providers and clinics to demonstrate use of evidence-based care. Structure measures could assess whether providers have systematic processes for training and supervision of staff, data systems for monitoring symptoms and functioning, and protocols for delivering evidence-based treatment and adjusting treatment based on outcome measures.

• Track use of psychotherapy. There are several limitations to using claims data to measure the quality of psychotherapy, but these data could be useful for tracking service use. Health plans and others could use claims to identify individuals with specific diagnoses or those receiving medications to examine their contact with the mental health care system. This type of a rudimentary measure would not assess the quality of care, but could provide insight into service utilization patterns that merit further investigation.

• Continue to refine measures and data systems for reporting. As measures are implemented, there will need to be ongoing efforts to maintain and refine them. For outcome measures, such refinement could address the selection of tools, challenges from statistical artifacts such as regression to the mean, risk adjustment, and other ways to account for differences in patient populations. Evidence-based processes of care could get incorporated into EHRs and claims; for example, EHRs could incorporate fields for reporting the delivery of specific treatment elements, and state Medicaid programs or health plans could use new billing codes to reimburse specific processes of care with strong evidence. This would require considerable investments in the data collection infrastructure, consensus-based efforts by multiple stakeholders to standardize billing codes and EHR fields, and efforts to support widespread adoption--all of which may be longer-term endeavors that rely on accumulating stronger evidence that specific processes of care are associated with outcomes. Structure and process measures would not replace outcome measures, but they may offer guidance on best practices and aid in quality improvement.

Quality measures are one tool that can improve psychotherapy outcomes. To have the greatest impact, a strategy for developing and implementing such measures should account for the strengths and limitations of different types of measures as well as opportunities for aligning quality improvement efforts and delivery system reforms. Support from a wide range of stakeholders will be needed to advance the implementation of measures that seek to improve the quality and outcomes of psychotherapy.

I. Introduction

A. Policy Context

Several types of psychotherapy can benefit individuals with the most prevalent mental disorders, including depression, anxiety, and post-traumatic stress disorder (PTSD). These psychotherapies include cognitive behavioral therapy (CBT), interpersonal therapy (IPT), psychodynamic therapy, and several others. Although the strength of the evidence supporting each varies, these therapies have generally demonstrated positive outcomes for adults and children (Jakobsen et al. 2012; Cape et al. 2010). Several clinical guidelines for the treatment of depression, anxiety, and PTSD recommend psychotherapy either alone or in combination with medications (National Institute for Health and Clinical Excellence 2005, 2009, 2011; American Psychiatric Association 2006).

As psychotropic medications have become widely used to treat common mental disorders in recent decades, data from national surveys suggest that the use of psychotherapy either alone or in combination with medications has decreased substantially (Olfson and Marcus 2010). Studies have found that only half of treatment episodes for depression include psychotherapy (Horvitz-Lennon et al. 2003), and fewer than 25 percent of older adults with depression receive psychotherapy (Wei et al. 2005). As few as 6 percent of veterans receiving outpatient care for PTSD receive evidence-based psychotherapy in the first six months of treatment (Shiner et al. 2013). Studies also suggest that few people receive a course of psychotherapy that is consistent with clinical trials; 40 percent of people do not return for their second psychotherapy visit, and fewer than 25 percent complete five therapy sessions (Simon et al. 2012a). Although there are no national data that point specifically to the content of psychotherapy and how it relates to protocols tested in research settings, these and other studies suggest that the majority of individuals with mental disorders are unlikely to receive psychotherapy consistent with the treatments tested in clinical trials.

Several factors may impede the receipt of evidence-based psychotherapy, including: (1) a lack of financial incentives to encourage quality and outcomes in psychotherapy; (2) inadequate training in evidence-based psychotherapies; (3) providers' attitudes toward the adoption of evidence-based treatments; and (4) barriers related to consumers' ability to access and sustain participation in psychotherapy. Although Medicare, Medicaid, and commercial insurers cover various forms of psychotherapy, in many states and communities the reimbursement is based largely on the number of visits or amount of time devoted to care rather than on the actual content (e.g., whether a specific evidence-based protocol was followed) or outcomes of the treatment (Weisz et al. 2013). It is important to note that many different types of providers--psychiatrists, psychologists, social workers, and other types of counselors--deliver psychotherapy in a wide range of treatment settings, including primary care offices, public sector mental health clinics, and small or solo private practices. Many of these providers are not affiliated with larger health care delivery systems that can offer clinical support and training on how to deliver care that is consistent with the latest research. These providers come from a variety of training programs and clinical orientations, and many have not received training in specific evidence-based psychotherapies (Arean et al. 2012). A frequently cited survey of accredited training programs in psychiatry, psychology, and social work in the United States found that very few programs required both didactic training and clinical supervision in common evidence-based therapies (Weissman et al. 2006). In addition, the requirements for provider certification to receive reimbursement differ across states, Medicaid programs, and health plans. In some states and communities, there are minimal standards that often rely on state licensing rules and providers' self-reported education and credentials without verification or rigorous continuing education requirements. Finally, research has found that mental health providers vary in terms of their willingness to adopt new treatment approaches that are inconsistent with their existing clinical orientation or that could threaten their autonomy (Aarons et al. 2009, 2011, 2012). Compounding these issues, many consumers face financial barriers to accessing and sustaining their participation in psychotherapy, including high-deductible health plans and limits on the number of covered psychotherapy visits. They may also discontinue therapy before achieving its full benefits due to their time constraints related to employment and family obligations.

Efforts to systematically monitor the delivery of psychotherapy are hindered by the fact that there are few widely accepted clinical quality measures that assess the receipt of psychotherapy, the degree to which it resembles the psychotherapy tested in trials, or the outcomes of care. We conducted an extensive review of behavioral health quality measures in early 2012 and found that most measures of psychotherapy focused on access to care or quantifying the number of visits rather than on the content or outcomes of care (Brown and Scholle 2012). For example, many measures were simple counts of visits or assessed whether individuals received a combination of psychotherapy and medications after an initial diagnosis. Such measures most often are calculated using data from claims or medical records. Of the psychotherapy measures identified none are endorsed by the National Quality Forum (NQF) (Brown and Scholle 2012; NQF 2014). Although Medicaid is the single largest payer of mental health services in the United States, a recent search of the U.S. Department of Health and Human Services (HHS), Centers for Medicare and Medicaid Services (CMS) Measures Inventory ("measures used by CMS in various quality, reporting and payment programs") identified no "psychotherapy" measures currently in use by CMS (CMS 2014). Psychotherapy measures are needed to inform quality improvement initiatives and to enable stakeholders to assess whether the services purchased were delivered as intended and achieved desired outcomes (Schoenwald 2011c; Bond et al. 2011).

Several measures have been developed to assess the delivery of psychotherapy for the purposes of distinguishing it from other treatment in efficacy trials; some of these measures are also used for training or ongoing clinical supervision (Schoenwald et al. 2011a, 2011b, 2011c). Such measures have not been widely adapted for the purpose of holding providers, health plans, or state agencies accountable for care or broader quality improvement initiatives. Many of these measures require extensive collection of observational data, such as videotaped sessions coded by trained raters, which may not be feasible in large-scale quality monitoring and improvement efforts.

Due to reforms being implemented under the Affordable Care Act (ACA) there is an increased need for practical measures to monitor the delivery of and outcomes of care. In particular, the ACA encourages health care delivery system reforms that use budgeted payments coupled with performance measurement to disincentivize inefficient use of health care resources counterbalanced with accountability for providing access to good quality care. In this regard, health homes and Accountable Care Organizations (ACOs) may present opportunities to improve the accessibility of psychotherapy while pay-for-performance efforts could incentivize the delivery of psychotherapy that achieves positive outcomes (Bao et al. 2013). However, some of these same reforms create pressure to limit access to care in order to control costs. Furthermore, due to Medicaid coverage expansions authorized by the ACA and through the health insurance marketplaces, demand for mental health services is expected to increase, and we may expect that more people will begin to receive various forms of psychotherapy. Understanding the extent to which the delivery of psychotherapy changes in response to these and other reforms requires a strong set of measures.

Several national quality reporting initiatives may present opportunities to include psychotherapy measures. These include states' voluntary reporting of quality using the Medicaid Adult and Children's Core Set and the Medicare and Medicaid Electronic Health Record (EHR) Incentive Programs for adoption and "Meaningful Use" of EHRs. In addition, the National Behavioral Health Quality Framework developed by the HHS Substance Abuse and Mental Health Services Administration (SAMHSA) provides momentum to advance the development of psychotherapy measures that can be used for quality improvement and accountability. Specifically, the development of quality measures focused on psychotherapy is consistent with the first goal of the SAMHSA quality framework--to promote the most effective prevention, treatment, and recovery practices for behavioral health disorders.

B. Objectives

We intend for this paper to provide a foundation for future measure-development activities. We briefly describe some of the evidence supporting the most common psychotherapies and discuss three types of measures: (1) structure measures, which gauge the capacity of providers and health systems to deliver evidence-based psychotherapy; (2) process measures, which assess the delivery of psychotherapy, including whether the content and duration of psychotherapy resembles the treatment tested in efficacy trials; and (3) outcome measures, which assess improvements in symptoms and functioning among individuals receiving psychotherapy. For each type of measure, we discuss its potential benefits and the feasibility of using different data sources that could support it, focusing particularly on the extent to which each type of measure would inform quality improvement efforts for a broad group of stakeholders relative to its potential data collection and reporting burden (Teague et al. 2012). We then propose short-term, medium-term, and long-term opportunities for developing and implementing quality measures addressing psychotherapy.

We sought to identify the strengths and limitations of different measurement approaches. Although we concentrate on the delivery of psychotherapy for common mental disorders of varying severity (depression, anxiety, and PTSD), the framework and findings are likely to be relevant for other psychosocial interventions that may benefit different populations. In addition, although the paper is not a comprehensive systematic review of the strength of the evidence base for psychotherapy, we have described some of the research and clinical guidelines that support psychotherapy to provide context for our discussion of various measurement approaches. Finally, the perspectives of a diverse technical expert panel of consumers, performance measurement experts, providers, health plans, payers, and state officials informed the development of this paper.

II. Common Evidence-based Psychotherapies

Although this paper is not intended to serve as a comprehensive or systematic review of the strength of evidence for each type of psychotherapy, this chapter provides a brief orientation to three major types of psychotherapy--CBT, IPT, and psychodynamic therapy--to provide context for the discussion of measurement approaches. We focus on these three because they are the most commonly mentioned therapies in treatment guidelines. The research supporting each type of therapy varies from randomized controlled trials (RCTs) to those that examine the effects of therapy without a control group (Clark et al. 2012; Driessen et al. 2010). This chapter also briefly discusses some of the shared features of psychotherapies and gaps in research that may impede the development and use of quality measures.

A. Cognitive Behavioral Therapy

CBT is a structured, focused, short (typically 6-16 weeks) therapy in which the provider and consumer work together to identify and address problematic thoughts and beliefs and their relationship with behaviors. The provider uses a range of techniques to alter these thoughts and behaviors, such as asking open-ended questions to help the individual recognize and challenge these thoughts and assigning homework that requires the individual to identify thoughts associated with particular events and the consequences of those thoughts (Beck 2011). CBT is goal oriented and tends to focus on the individual's current problems. CBT is one of the most widely used psychotherapies; in a survey of mental health professionals, 79 percent of respondents identified CBT as one of their theoretical orientations (Cook et al. 2012).

Studies suggest that CBT is effective in treating depression, anxiety, and PTSD in both children and adults (Cape et al. 2010; Chambless and Ollendick 2001; Clark et al. 2012; Foa 2009; Forman-Hoffman et al. 2013; Hofmann et al. 2012; Institute of Medicine 2012; Jonas et al. 2013; Otte 2011). In Table II-1, we provide a snapshot of clinical guidelines that recommend CBT. Most of these guidelines include a recommendation for an initial treatment length that varies from 8-12 sessions for adults with PTSD and 16-20 sessions for adults with major depression. These guidelines generally recommend CBT based upon the results of randomized and non-RCTs. There is some evidence supporting the effectiveness of CBT in treating attention deficit hyperactivity disorder in adults (Ramsay 2007). Although the majority of research on CBT has examined its use in specialty mental health treatment settings, a recent meta-analysis supports using CBT in treating depression and anxiety in primary care settings (Cape et al. 2010).

B. Interpersonal Therapy

IPT is newer than CBT and has been used predominantly in the treatment of depression. It is provided over the course of 12-16 weeks and focuses on the connection between an individual's mood and interpersonal stress (Markowitz and Weissman 2004). IPT providers address the context in which the presenting problem(s) arose and help the individual to develop coping and communication skills, and to improve his or her relationships with others.

The research supporting the effectiveness of IPT is not as extensive as that for CBT, but there is some evidence for its effectiveness in treating depression among both children and adults. The National Institute for Health and Clinical Excellence (2009) recommended using IPT in the treatment of depression and suggests that individuals should initially receive 16-20 sessions of IPT over a 3-4 month period. Two other guidelines also recommend using IPT in the treatment of depression (see Table II-1), but do not suggest a specific length of treatment. Studies suggest that IPT is more effective than usual care or wait list controls, and some studies have shown that it is as effective as CBT in treating depression (Clark et al. 2012; Cuijpers et al. 2011; Jakobsen et al. 2012; Maalouf and Brent 2012; Van Hees et al. 2013). The effectiveness of IPT has been demonstrated in specialty mental health treatment settings, but its effectiveness in primary care settings is unclear (Cape et al. 2010). Its effectiveness in treating mental health problems other than depression is also uncertain.

C. Psychodynamic Therapy

Psychodynamic therapy is intended to help the individual build awareness and understanding of his or her behaviors, thoughts, and feelings, and on how past experience relates to current functioning (Shedler 2010). Psychodynamic therapy is traditionally delivered over a relatively long period of time, sometimes years; recently, however, there has been greater interest in short-term psychodynamic therapy, which is generally a maximum of 40 sessions (Leichsenring 2005).

Although one survey of mental health professionals found that 36 percent of the respondents reported using psychodynamic therapy (Cook et al. 2012), relatively few studies have rigorously evaluated its effectiveness. Some of the studies have small samples and vary in the rigor of the design, but there is evidence to support the use of psychodynamic therapy in treating depression (Abbass and Driessen 2010; Leichsenring 2005; Leichsenring and Leibing 2007). The results from a meta-analysis suggest that psychodynamic therapy used in combination with medication produces better outcomes than medication alone (Jakobsen et al. 2012), and some studies suggest that psychodynamic therapy is equally effective as CBT in treating depression (Leichensring and Leibing 2007). Although the literature on the effects of psychodynamic therapy on other common mental disorders is limited, some evidence points to its efficacy in treating some types of anxiety (Abbass and Driessen 2009; Leichsenring 2005). As summarized in Table II-1, five of the eight clinical guidelines include psychodynamic therapy; however, they range from recommending psychodynamic therapy with "moderate clinical confidence" to concluding that the "balance of benefits and harm is too close to justify a recommendation." The field would benefit from additional research on the effectiveness of different types of psychodynamic therapy on depression, anxiety, PTSD, and other common mental disorders.

D. Summary

The evidence to support the effectiveness of specific psychotherapies varies. The evidence for IPT and psychodynamic therapy focuses mostly on depression and anxiety; CBT has been used more widely and tested in more populations, including people with ADHD and PTSD. Although this chapter briefly reviews three psychotherapies, there are several other types of therapy that may benefit consumers but have undergone less empirical testing. Some researchers have noted the lack of information on the processes and outcomes of psychotherapy in typical treatment settings and have suggested that further research into usual care practices may identify promising treatment approaches (Garland et al. 2010a; 2010c). The variability in the evidence for different types of psychotherapy, as well as the limited research on the effectiveness of some types of psychotherapy, has several implications for quality measurement. As with the development of any quality measure used for accountability, the measure developer and user would need to assess the strength of the evidence for a particular psychotherapy when applied to a particular population. Further, much of the evidence on psychotherapy has been conducted in controlled settings. There is limited evidence on the extent to which a particular psychotherapy can be adapted while still achieving positive outcomes.

We also recognize that some of these psychotherapies share common elements (Chorpita and Daleiden 2009). There is a large body of literature to suggest that common factors and common elements across different types of psychotherapy are strongly associated with outcomes independent of the type of treatment being delivered (Laska et al. 2013; Lambert and Barley 2002). Common factors include the ability of the provider and consumer to agree on goals and collaborate toward reaching those goals, the provider's ability to demonstrate empathy, and the development of a strong therapeutic alliance (that is, the quality of the bond with the therapist and whether the therapist shares his or her therapeutic objective) (Bickman 2005). Common treatment elements are the shared therapeutic features of different types of psychotherapy. These include elements such as the use of relaxation techniques, psychoeducation, and cognitive coping skills (Laska et al. 2013). For example, a review of evidence-based interventions for treatments of anxiety in children found that most offered psychoeducation, exposure, and relaxation (Chorpita and Daleiden 2009). Thus, although this chapter describes some of the specific types of psychotherapy as targets for quality measures, it could be possible to measure whether some of the common factors and treatment elements are present and result in positive outcomes.

The current evidence base on psychotherapy has not given sufficient consideration to the needs of people with co-occurring disorders. Many studies of these psychotherapies have focused on single diagnostic groups, so the outcomes of psychotherapy for people with multiple mental health problems or both mental health and substance abuse problems is often unclear. Existing clinical guidelines do not provide consistent recommendations on the treatment of individuals with co-occurring behavioral health disorders or those with comorbid physical health conditions.

Finally, disparities in care based on race and ethnicity have raised concerns about the need for adapting evidence-based treatments to specific cultural contexts and beliefs (Sue et al. 2009). Guidelines about cultural competency in treatment approaches recommend that providers consider such issues as cultural beliefs that may pose barriers to care or contribute to differences in expressing symptoms (for example, AACAP 2013); however, the question of whether evidence-based treatments should be adapted to specific cultural needs is still debated (Bernal et al. 2009). Additional research on the effectiveness of psychotherapies among diverse populations would be beneficial.

III. Evidence-based Psychotherapy in Practice and Options for Quality Measures

Efficacy trials of psychotherapy typically include a certain number of visits or an expected progression through phases of treatment. As psychotherapies move from efficacy trials to practice, the content and duration of treatment is often adapted to the provider's clinical orientation or the context of the treatment setting. In addition, treatments may be tailored to the consumer's illness severity, past experience with treatment or other characteristics. Adhering to the treatment protocols tested in trials may be challenging for a variety of practical reasons.

There is also strong evidence that mental health providers tend to adopt "eclectic" treatment approaches that draw from different theoretical orientations and professional training (Garland et al. 2010a). Rarely does a provider only offer one type of psychotherapy (CBT, for example) and the provider may blend techniques from different types of psychotherapy to meet the consumer's needs. There is also evidence that providers tend to use some therapeutic skills more than others. For example, one study that videotaped psychotherapy sessions of children found that providers delivered some key elements of treatment frequently (for example, positive reinforcement and psychoeducation) but other components were less common (for example, role-playing or assigning homework) (Garland et al. 2010b). Another study found that when providers were asked about their perceptions of cognitive therapy before they learned how to use it, many of them reported that they planned to selectively implement elements of the therapy rather than attempt to deliver the full protocol (Stirman et al. 2013). Consequently, the care delivered in typical community-based treatment settings might not resemble the psychotherapy tested in trials (Brookman-Frazee et al. 2008, 2010; Baumann et al. 2006). Such adaptations are understandable and may facilitate the wider dissemination of the treatment (McHugh et al. 2009; Chorpita and Regan 2009), but researchers do not have a good understanding of how specific adaptations in the duration or content of psychotherapy affect outcomes. Although there is good evidence to suggest that the duration of treatment in the community is rarely consistent with trials (Simon et al. 2012a), there is very little information on the content of psychotherapy delivered outside of research settings (Garland et al. 2010c).

When care in the community has been studied, researchers have noted that evidence-based psychotherapies implemented in typical community-based treatment settings produce more modest outcomes than efficacy trials (Weisz et al. 2013; Weisz et al. 2006). This may be because care in the community diverges from the protocols tested in trials; it may also relate to the broader population of consumers included in community-based treatment settings (where efficacy trials tend to have many exclusion criteria). This observation has several implications for quality measurement. Efforts to advance the adoption of evidence-based psychotherapies are likely to require substantial changes in practice for many providers, payers, and consumers. Making these changes is likely to require intensive support for provider training and consumer engagement as well as new payment models that promote evidence-based practice. Confidence in the anticipated outcomes of psychotherapy is critical to both the successful implementation of quality measures and the spread of evidence-based psychotherapy.

To guide our consideration of the possible approaches to measuring the quality of psychotherapy for the purposes of accountability and quality improvement, we used the Donabedian (1980, 1988) quality framework to describe measures of structures, processes, and outcomes. In Table III-1, we summarize the measure concepts, potential data sources, and strengths and limitations of each type of measure. Structure measures evaluate the organization--including such resources as the staffing, equipment, and protocols. For psychotherapy, structure measures would focus on whether providers have the capacity to deliver evidence-based care and to monitor outcomes--for example, whether providers have appropriate training, use treatment protocols and tools consistent with psychotherapies tested in trials, and monitor symptoms and functioning using standardized tools. Process measures describe how structures are put into place and are quantified for consumers (for example, process measures would allow us to determine the extent to which individuals received psychotherapy that is consistent with the content of treatment that tested in trials). Because treatments are adapted to fit into service settings, process measures tell us something about the extent to which these adaptations result in the delivery of treatment that still contains the core components of the evidence-based model. Finally, outcome measures assess the end results of processes; such measures would examine whether the delivery of psychotherapy reduces the severity of a consumer's symptoms and improves his or her functioning.

IV. Structure Measures

Structure measures assess the capacity of a provider, clinic, or health care organization to deliver evidence-based care, and they may address such topics as staffing, data systems, and treatment procedures (Donabedian 1988). Structure measures are most often used in accreditation or certification programs that set minimal requirements for health care organizations (AHRQ 2013a) and may be helpful as a roadmap for organizations to follow when they assume responsibility for new activities or new populations (Teague et al. 2012; NCQA 2013).

With current federal efforts to spur delivery system reform, structure measures are being used to qualify organizations for participation in programs or eligibility for new incentives. For example, consistent with the ACA's incentives for developing new models of care and adoption of health information technology, structure measures have been introduced for behavioral health homes that take on responsibility for integrated mental and physical health care (SAMHSA-HRSA Center for Integrated Health Solutions 2012). To receive financial incentives for meaningful use of EHRs, providers are required to meet objectives for collecting and using particular types of data. Structure measures may also be used to certify providers, thereby making them eligible for new payment arrangements. For example, in some states, Medicaid providers must demonstrate that they meet certain standards before they can bill for Multisystemic Therapy (North Carolina MST Funding and Medicaid Standards n.d.).

In mental health care, structure measures are among the diverse means for gauging the fidelity of treatment (Teague et al. 2012). Fidelity can be conceptualized as a combination of provider competence in delivering treatment, provider adherence to treatment, and the extent to which the treatment differentiates itself from other treatments (Schoenwald and Garland 2013; Schoenwald et al. 2011a, 2011b). Because treatments are adapted to fit into service settings, structure measures can assess whether organizations have the capacity to deliver evidence-based psychotherapy (sufficient numbers of trained staff and procedures for supervision, for example) while process measures indicate something about the extent to which the adaptations result in the delivery of treatment that still contains the core components of the evidence-based model. Pairing structure and process measures may help to provide a more comprehensive understanding of the quality of care. For example, a study of mental health care in the U.S. Department of Veterans Affairs (VA) used a facility survey to assess the availability of evidence-based practices such as CBT as well as process measures of the receipt of evidence-based practices (the percentage of patients with PTSD who had any CBT visit during the study period) (Watkins et al. 2010, 2011).

Structure measures to assess the capacity of clinics or organizations to deliver evidence-based psychotherapy could address such topics as:

• Availability of staff who are trained in evidence-based psychotherapy.

• Adoption and implementation of protocols or guidelines for the use of evidence-based psychotherapy.

• Availability of tools that support a consumer's engagement in therapy (web-based tools, workbooks, or homework materials).

• Availability and use of tools for assessing symptoms and functioning, and the use of those tools to monitor outcomes.

Likely data sources for structure measures include data reported by an organization or clinic through surveys, reports, or documentation. Audits or reviews of documentation may be warranted to ensure the validity of these reports when payment is at stake.

A. Advantages

Assesses the availability of services. Structure measures can provide critical information about the adequacy of the network of providers within a health system, health plan, ACO, or state in terms of their training and capacity to offer evidence-based psychotherapy. Such information is helpful for assessing gaps in accessibility of care, especially in rural or underserved areas.

Provide guidance on infrastructure development and best practices. By defining the infrastructure needed to provide evidence-based psychotherapy, structure measures could help clinics and organizations understand and implement new approaches to care. As described in a later chapter, these new approaches to care could include the use of repeated outcome assessments to report symptoms and functioning to guide care. These measures could also have the flexibility to address different types of psychotherapies that would be relevant to different populations or provider settings. Structure measures could build on existing approaches to fidelity training and supervision used in research and developed for specific psychotherapies. Similarly, structure measures could lay out expectations for data systems and protocols for collecting information on symptoms and functioning and how to adjust treatment when there is no improvement. Such an approach could offer flexibility to select tools that are relevant to different populations or conditions.

Support credentialing and payment. Structure measures could support or complement existing credentialing processes, thereby allowing purchasers and health plans to select clinics or provider organizations that are equipped to provide evidence-based psychotherapy services and outcomes monitoring. Alternatively, performance on structure measures could be used as the basis for differential payment and incorporated into new models, such as the behavioral health home.

Support consumer choice. Health plans or other entities could make information available to consumers about the capability of providers to allow them to select providers that have expertise in evidence-based psychotherapy, including treatments specific to their conditions. This information could be provided in health plan provider directories and also made available to other providers to facilitate referrals to care consistent with consumers' desired treatment.

B. Challenges

Lack of clarity on necessary structures. Defining structure measures that are clear but also flexible enough for implementation in different settings can be challenging. Although there is evidence to support the use of certain psychotherapies, research has not identified the structures that contribute to psychotherapy outcomes (for example, addressing the matter of whether there is an evidence base for the amount of training or supervision needed to implement the treatment with fidelity in typical treatment settings).

Limited resources for investing in training and infrastructure development. Building the capacity to deliver evidence-based psychotherapy may require significant investments in staff time as well as outlays for expert training, consultation, and supervision as therapists learn and implement new skills. This is especially a concern for proprietary therapies. Building data systems to support outcomes monitoring may also represent new expenses for mental health providers, particularly given the lower rates of EHR adoption in these settings (Druss and Dimitropoulos 2013).

Documentation burden. Documenting the steps involved in implementing evidence-based psychotherapies or use of structured processes for monitoring outcomes may impose an additional burden on clinics or organizations, and reviewing this information with an external auditor adds to costs. Managed care organizations and states now credential providers based on licensure, and they typically obtain self-reported information on expertise and training. New efforts to certify health homes offer an avenue for implementation, but auditing and documenting structures described here would be a new activity. Because of restrictions on access to behavioral health records in some locations, auditors may have to make special arrangements with provider organizations, health plans, or state agencies to access documentation for external review. It is also unclear whether the proprietary nature of some psychotherapies would limit the kinds of standards set for supervision or training.

Although audits could focus on protocols and reports, they might also include using independent raters to assess whether providers demonstrate competency in delivering evidence-based psychotherapy. Many studies have used trained, independent raters who either directly observe the delivery of therapy or review video or audio of therapy. Independent raters are considered the most valid reporters of fidelity to psychotherapy because they have been specifically trained to recognize the delivery of the therapeutic elements and, because their rating has no personal bearing for them, their responses are the least likely of all raters to be biased (Barber et al. 2007; Schoenwald et al. 2011a). Clinics or organizations could contract with independent raters as part of a system of evaluating the competency of their providers (although clinics would need to take special care to protect privacy of consumers).

Credentialing on the basis of reports from independent raters could be burdensome and costly. Nonetheless, some states are doing this. For example, beginning in fiscal year 2014, the Texas Department of State Health Services will require providers who offer psychotherapy to adults with depression to demonstrate competency in CBT, which will be determined by an independent rater who reviews and scores a videotaped therapy session (Robinson 2012). Providers who score below an established threshold will be prohibited from providing psychotherapy to adults with depression until their competence has been established. The state has given local mental health authorities a list of organizations qualified to provide independent competency ratings, which costs $150-$300 per videotape reviewed. As of this writing, we do not know how often providers must demonstrate competency, nor do we know the total cost to the Texas health system to obtain independent ratings.

C. Summary

Structure measures are appealing as an initial step for setting expectations related to how providers should implement evidence-based psychotherapies and monitor consumer symptoms and functioning over time. This is particularly useful given the very limited information on the use of evidence-based therapies in routine settings and concerns that where the therapies are implemented they are not being done with sufficient adherence to protocols demonstrated to be efficacious in research trials. Furthermore, as described in a later chapter, processes for outcomes monitoring are critical because even in randomized trials showing the benefit of evidence-based treatment, many participants do not benefit and efforts to adjust treatment are needed. Structure measures are likely to serve best as building blocks to guide the delivery of care and as complements to other measures.

V. Process Measures

Process measures assess whether individuals receive care or treatments that have evidence of improving outcomes. In mental health, several process measures address the management and continuity of effective medications (for example, the management of antidepressants and the continuous use of antipsychotic medications for schizophrenia). These measures are used in federal, state, and private sector reporting programs. In contrast, process measures that assess the delivery of psychotherapy have not been widely adopted. The measures listed below could, however, be adapted from existing measures:

• Receipt of evidence-based psychotherapy for specific indications (CBT for depression, anxiety, or PTSD, for example).

• Completion of recommended course of psychotherapy (based on completing encounters focused on specific content).

• Continuity of psychotherapy (based on number of visits over a given period for individuals for whom ongoing psychotherapy has been recommended).

• Consumer-reported or provider-reported content of psychotherapy.

Measures assessing the receipt of evidence-based psychotherapy would require information on diagnoses, severity of illness, and treatment history; consumer preferences for treatment; and the number, timing, and content of visits or sessions. Potential data sources for this information include claims, medical records (including EHRs or other health information technology), and surveys of consumers or providers. As explained below, each data source and type of measure has various strengths and limitations.

There may also be value in constructing measures that track the accessibility or encounters with the mental health care system but do not have evidence to support the impact of these services on outcomes. These "related health care delivery measures," as they are termed by the National Quality Measures Clearinghouse, include measures of service use and costs. For example, this might include measures of psychotherapy visits that did not specify the content of the visit or how it was appropriately targeted to a specific need of an individual. Our focus in this paper is on quality measures that can assess services with evidence of benefit on outcomes.

A. Measures Based on Claims

Measures based on claims would be limited to: (1) assessing whether consumers with an indicated need for treatment (that is, those with a specific diagnosis) receive any psychotherapy; and (2) counting the number of visits over a specific period of time.

1. Advantages

Claims data are low-cost/low-burden. Using claims data is a low-cost, low-burden approach to tracking the receipt of psychotherapy. Researchers have proposed such measures and demonstrated some link to improved outcomes (Rost 2005). Some measures of continuity of care have demonstrated a strong positive association with outcomes among people with serious mental illness (Adair et al. 2005). Data from health plans and the VA health system suggest that the number of visits for most individuals is not concordant with guidelines and is affected by distance to the site of care (Pfieffer 2011); availability of trained providers is also likely to be a factor. Such a measure could be useful in monitoring possible disparities in access to care.

Such measures could be more readily implemented than the structure and outcome measurement approaches described here. They could be an interim step to ensure psychotherapy, an important form of behavioral health treatment, is delivered as part of broader health care system reforms as they are being implemented and then later updated as more advanced forms of performance measurement for psychotherapy are developed.

2. Challenges

Billing codes for psychotherapy lack detail on the content of visits, and they vary across health plans and states. The lack of specificity in psychotherapy claims contrasts with measures that look at the course of medication treatment (continuity of antipsychotics for people with schizophrenia, for example) where there is detailed information on the active ingredient and dosage that was filled by the consumer (although one cannot be certain that medications are actually taken). Currently used billing codes for psychotherapy have broad labels like "individual psychotherapy" or "group psychotherapy," and they provide no detail on what psychotherapeutic techniques were used or what content was delivered during the visit. Although ICD-10-PCS codes for procedures do have more detailed information (for example, 2014 ICD-10-PCS Code GZ58ZZZ, http://www.icd10data.com/ICD10PCS/Codes/G/Z/5/8/GZ58ZZZ, Individual Psychotherapy, Cognitive-Behavioral), this procedure classification is slated for use in the United States for procedures performed in hospital inpatient settings and would have limited applicability to ambulatory care (American Psychological Association 2013; CMS 2013). In addition, state Medicaid programs (and other state-based programs) have developed their own psychotherapy billing codes (some state Medicaid programs have hundreds of such codes). Some of these codes are tied to the credentials of the provider, but few give detail on the content of the psychotherapy. Such variation in the billing practices of state Medicaid programs complicates the use of claims data to measure the content of psychotherapy and make valid comparisons (Brown et al. 2012).

Current practices for coding diagnoses also make it challenging to define the eligible population for specific psychotherapy. The introduction of ICD-10 diagnoses codes (planned for 2014) will provide more detailed coding that will make it possible to target eligible population and to improve coding (CMS 2013). For example, codes will allow for documentation of exacerbating features and status of remission. However, claims data are limited to services provided and covered by a specific payer. Information on care is limited to the period of enrollment, and care provided in other settings (for example, employee assistance programs, schools, or state-funded mental health clinics) may not be captured.

The need for a clear link between billing codes and outcomes is a fundamental criterion for national endorsement of quality measures. There may be opportunities to develop new billing codes to capture data on the use and content of psychotherapy, but changes in payment policies are likely to be needed to encourage providers to implement and consistently bill for such services. CMS has recently introduced new billing codes for evidence-based behavioral health services, such as depression screening and Screening, Brief Intervention and Referral to Treatment for alcohol misuse, but at this time it is unclear how widely these new codes are used and what changes in Medicare payment policy will be necessary to encourage their widespread use.

Promoting visits, not quality. Measures that promote visits while ignoring the content of psychotherapy and outcomes have the potential to encourage service utilization without proof of its benefits. To date, there is little evidence that mental health services are being overused (Druss 2007); however, it may be difficult to measure adequately from claims. Examples of issues that are likely to arise during measure-development include what type of contact between a provider and consumer is necessary to constitute evidence-based care (that is, in person or by phone, or with a primary care provider or behavioral health specialist) and what duration of treatment or number of visits is necessary. The frequency and duration of treatment would be informed by psychotherapy trials, but there are many patient-level characteristics that can affect the treatment course but that are not captured in claims data, such as illness severity and experience with past treatment. Finally, there is little evidence that shorter episodes of psychotherapy are necessarily associated with worse outcomes. Some research has even suggested that consumers who drop out after a single psychotherapy visit may have the most favorable outcomes (as measured by self-reported clinical improvements) and are highly satisfied with care, but further research is necessary to understand these patterns (Simon et al. 2012b; Baldwin et al. 2009; Stiles et al. 2003). Claims-based measures assessing the duration or number of psychotherapy visits may be most useful in the context of matched clinical outcomes data.

3. Summary

Measures that assess the delivery and continuity of psychotherapy based on claims data may be reasonable as interim measures that provide a crude picture of service utilization. It is important, however, to consider the potentially negative consequences of encouraging care with unknown benefits. Measures based on claims could be more useful if billing codes were improved and if billing were tied to credentials or certification; that is, if the billing code indicated that trained professionals were providing specific psychotherapeutic content.

B. Measures Based on Medical Records (including electronic health records and other data)

Compared with measures based on claims data, measures that assess receipt of a recommended number of visits with specific psychotherapeutic content for an episode of mental illness would be more strongly linked to evidence. These measures could be developed if information on the delivery of specific psychotherapeutic content (for example, whether psychoeducation was provided or homework was assigned) were documented in medical records, particularly in EHRs that have structured fields or in other electronic data reporting systems. In addition, some therapies can be provided or aided by health information technology (Mohr et al. 2013). Web-based interventions have shown efficacy for a variety of conditions. The interventions may include didactic materials as well as interactive tools that allow individuals to track activities or distress and get feedback on their progress. Therapists have also used web-based programs to support delivering evidence-based treatment (Craske et al. 2009). These programs may be able to track the content and duration of engagement in evidence-based treatment. By incorporating clear details on the content of therapy, these data sources could be useful both for tracking the receipt of evidence-based care and for quality improvement.

1. Advantages

Low cost if necessary data elements are available electronically. Measures developed from EHR or other health information technologies would allow information on the content of therapy to be extracted more readily. Although EHRs hold promise, adoption of such technology is lagging in mental health treatment settings (Druss and Dimitropoulos 2013), and they often do not contain the necessary fields to capture psychosocial health factors and the specific psychotherapeutic content of visits (Glasgow et al. 2012). Computer algorithms to abstract free text notes in EHRs to identify whether evidence-based psychotherapy was provided have been developed (Shiner et al. 2013); however, these methods have undergone limited testing, and current quality measurement paradigms for EHRs continue to depend on structured fields. Although tracking the duration of use and content viewed in web-based interventions sounds attractive, the feasibility of extracting such information in reliable ways is unclear.

2. Challenges

Confidentiality. Even when there are no legal impediments to sharing data for quality measurement, current practices for protecting the confidentiality of consumer records limit the feasibility of collecting detailed information on the content of care. Health plans and other entities will need to make special efforts to assure consumers and providers about the confidentiality of mental health or substance abuse records.

Additional documentation required from providers. Efforts to capture information on the delivery of evidence-based psychotherapies could be incorporated into medical records, but this would require changes in practice and policy. Because of confidentiality concerns and feasibility issues, prospective reporting in electronic tools (EHR or other systems) would be important to successful implementation. Systematically recording specific therapies or psychotherapeutic content of visits would represent a major change in documentation practice for providers. In organizations where it is common to have limited notes on specific visits, this additional documentation may not be well accepted. Certain forms or tools might alleviate the burden associated with the additional documentation, but this might result in "checkbox" measures that would be easy to implement but might warrant special effort for auditing or review.

C. Consumer and Provider Reports of the Content of Psychotherapy

Measures of whether specific psychotherapy content was delivered could be based on data collected through consumer or provider–reported surveys. When used for accountability purposes, such measures would have to consider the optimal timing and recall periods and the ability of consumers and providers to accurately report the content of treatment. In this section, we describe the strengths and limitations of measures that require consumers or providers to report the content of treatment; in the next chapter we discuss consumer-reported outcomes.

1. Advantages Associated with Consumer-Reported Measures

Consumer-reported measures provide an opportunity to assess the delivery of care from the perspective of the person receiving and ultimately impacted by the care. Such measures could also provide a mechanism for consumers to engage providers in discussions about how treatment will be provided and the rationale behind the treatment. For example, completing a measure at some point during treatment may encourage the consumer to ask for certain components of treatment (homework assignments, and so on), which could change the course of treatment. Such discussion between the consumer and provider may strengthen the therapeutic relationship.

Consumers are routinely surveyed to assess their satisfaction with services or their broader experiences with care (using, for example, measures such as the Consumer Assessment of Health Care Providers and Systems (CAHPS^®) surveys and the CAHPS^® Experience of Care and Health Outcomes survey for behavorial health care). Consumer-reported measures that attempt to assess the specific content of psychotherapy visits are less common. Miranda and colleagues (2010) have taken initial steps to develop a consumer-reported measure that assesses the use of evidence-based psychotherapy for depression. They developed survey items based on existing adherence and competence measures, on core therapy components identified in a review of literature, and in consultation with clinical experts in CBT, IPT, and psychodynamic therapy. They refined the measure using feedback from consumers who participated in cognitive interviews. They tested the measure in a sample of consumers who received services through a managed behavioral health care organization. Consumers were eligible to complete the survey if they had at least three outpatient psychotherapy visits for major depression in the prior six weeks. The consumers retrospectively rated the frequency with which their provider delivered each treatment element over the course of their treatment. The preliminary results suggest that the measure is reliable. The authors also reported that the consumer's reports on the provider's use of psychodynamic or cognitive behavioral approaches were associated with the provider's own identification with these therapeutic orientations, which the authors suggest provides preliminary evidence of the measure's validity. The study represents an important first step in developing a consumer-reported survey that assesses the fidelity of psychotherapy delivered to adults with depression. As suggested by the authors, the measure should be further tested in studies of CBT, IPT, and psychodynamic therapy.

2. Challenges Associated with Consumer-Reported Measures

There is some research to suggest that consumers may not be valid reporters for measures of the content delivered in psychotherapy. Two studies that examined reports from youth and their caregivers on their provider's delivery of mental health and substance abuse treatment found that they almost always reported that the provider delivered the therapeutic elements and that there was little agreement between their ratings and those by providers, clinical supervisors, and independent raters (Chapman et al. 2013; Schoenwald et al. 2009). Schoenwald et al. (2009) found that the consumer-reported ratings did not significantly vary over time. The researchers suggest that this could indicate that consumers may not be able to detect variation or changes in the delivery of specific psychotherapeutic content.

Consumers may have difficulty recalling therapy sessions. Respondents could complete surveys that collect data on the content of a single therapy visit, at multiple points in time, or retrospectively across multiple visits (Barber et al. 2007). The visits to be evaluated could be randomly selected or pre-determined. The greater the time lag between the therapy session and completion of the measure, the more difficult it may be for consumers to accurately report on the clinician's delivery of therapeutic elements.

Elements of psychotherapy change over the course of treatment. For example, providing information to the consumer about his or her problems or discussing expectations for treatment may be prescribed elements that occur at the beginning of therapy. In addition, the pace at which a provider delivers certain content may also vary from consumer to consumer, depending on the individual's rate of progress. Therefore, the timing and look-back period of a measure would have to account for these factors so providers or organizations are not inappropriately penalized for not delivering certain aspects of treatment or for not delivering them at the prescribed time. Asking the consumer to report the content of each visit directly after the visit might yield the most accurate assessment. Alternatively, a measure completed at a single point in time that attempts to capture therapy elements across several visits might address concerns about incorrectly penalizing a provider for not delivering certain elements at a certain time. Such an approach, however, would lend itself to recall problems.

Burden and costs of data collection. Introducing surveys to collect sufficient feedback from consumers to make reliable and fair comparisons of performance by individual providers or groups of providers would represent a new investment. Although on-site data collection might ease the cost of distribution, it could lead to bias (Anastario et al. 2010) (though bias might be able to be overcome as has been attempted in an on-site CAHPS^® survey) (see the NQF endorsed CAHPS^®Nursing Home Survey: Long-Stay Resident Instrument, NQF #0692). The use of electronic surveys could reduce the cost of data collection, especially if integrated into the EHR. However, newer methods of information collection (such as the use of patient portals or touch screen computers or tablets in the office) have not been well studied in psychotherapy.

3. Advantages Associated with Provider-Reported Measures

Provider self-reported measures of the content of psychotherapy have demonstrated promising results. Measures that ask providers to self-report the content of the psychotherapy they deliver have not been widely used outside of research settings or clinical supervision. Hepner and colleagues (2010), who worked with Miranda et al. (2010) to develop the consumer-reported measure of depression care described above, developed a parallel measure for providers. Providers were asked to select an individual currently in treatment for major depression and to rate the frequency with which they delivered each therapeutic element over the course of treatment. The results of testing the measure parallel the results for the consumer measure: there is support for the measure's reliability and preliminary evidence for its validity based upon positive associations between the reported use of psychodynamic or cognitive behavioral approaches and the providers' identification with these therapeutic orientations.

4. Challenges Associated with Provider-Reported Measures

The validity of provider-reported content of psychotherapy has not been fully established. Providers may over-estimate their delivery of specific treatment content, particularly if the measure is linked to performance appraisals or payment (Schoenwald et al. 2011b). Work by Martino and colleagues (2009) suggest that the validity of provider-reported measures may depend on the aspect of the psychotherapy being rated. They found that providers, supervisors, and external raters generally agreed on the provider's delivery of the basic elements of the particular treatment, such as the use of open-ended questions and affirmations; however, there was less agreement in their assessments of the providers' delivery of advanced psychotherapeutic treatment techniques. Providers rated their delivery of certain advanced components higher than did supervisors and external raters. Consistent with these findings, Chapman and colleagues (2013) found that providers over-estimated their ability to follow the treatment protocol when compared with independent raters.

Providers may have difficulty recalling therapy sessions. As mentioned previously, a measure of the delivery of psychotherapy could assess the care provided in a single therapy visit, at multiple points in time, or retrospectively across multiple visits. Just as with a consumer-reported version of the measure, the greater the recall period in completing the measure the more difficult it may be for providers to accurately recall the specific therapeutic elements delivered in a given session. As a result, providers may over-report the content of therapy delivered unless the information is recorded immediately following the session, perhaps as part of an EHR or some other electronic system.

5. Summary

Asking consumers or providers to report the content of psychotherapy may be burdensome. Although obtaining consumer perspectives on care is a well-established method for quality measurement, the validity of using consumer or provider surveys to report the content of psychotherapy for accountability purposes is unclear.

As mentioned above, an alternative to asking providers to report the content of visits through surveys or questionnaires would be to incorporate common fields in EHRs to capture this information. Asking providers about their usual treatment practices may be also more appropriate as part of a structure measure to assess competency.

VI. Outcome Measures

Outcome measures would assess whether individuals receiving psychotherapy experience improvements in their symptoms and functioning across a broad range of life domains, including employment, school functioning, relationships, and engagement in the community (Hoagwood et al. 2012). The use of such measures would complement efforts already underway to encourage the measurement of patient-reported outcomes in other areas of health care. For example, CMS and the Office of the National Coordinator for Health Information Technology are supporting efforts to develop patient-reported outcome measures for use in the future stages of the CMS EHR incentive program. These measures address medical and behavioral health conditions. In addition, HHS recently supported a NQF project that sought to identify the factors that should be considered for selecting patient-reported outcome measures for performance improvement and accountability (NQF 2013). Finally, the ACA established the Patient-Centered Outcomes Research Institute to conduct comparative effectiveness research focused on outcomes important to patients. Many of these projects employ or are testing measurement strategies that offer feedback to providers and consumers in an effort to support their ability to make treatment decisions.

As in other areas of health care, measuring psychotherapy outcomes could involve the use of repeated assessments to track improvements over time and progress toward reaching the consumers' goals. These assessments could be completed directly by the consumer. With consumer consent, family members or other individuals important to the consumer, such as case managers or teachers (particularly for children or adolescents receiving therapy), could also complete assessments since these reporters may offer different perspectives (Brown et al. 2008, 2007). Providers could use information from these repeated assessments to adjust treatment in response to an individual's progress, and the individual receiving treatment can use the information for self-monitoring and to make treatment decisions (for example, inquiring about more intensive treatment options or changing providers). Such data could be stored in medical records/EHRs or in other electronic systems, such as web-based systems. In the aggregate, health systems can use the data from these repeated assessments to monitor how consumers respond to treatment and to identify opportunities for quality improvement. Such data can facilitate comparisons of outcomes across these entities. This approach of using repeated assessments to measure outcomes, tailor treatment, and improve the quality of care--referred to as measurement-based care or routine outcomes monitoring in the literature (Boswell et al. 2013; Harding et al. 2011)--is common for physical health conditions, such as diabetes and hypertension, for which measurements are taken regularly and treatment is adjusted accordingly.

There are several examples of measurement-based care or routine outcome monitoring systems in mental health care (Drapeau 2012). Here we provide a brief description of some of these systems to illustrate how they work in practice.

Perhaps one of the most well-known outcomes monitoring efforts in mental health care is the Depression Improvement Across Minnesota Offering A New Direction (DIAMOND) project, in which participating health plans pay certified practices a flat monthly rate for providing a bundled set of services for depression or dsythymia. As part of the initiative, practices administer the PHQ-9 during the consumer's first visit and again at six months and 12 months after the initial visit (AHRQ 2013b). Practices receive monthly performance reports that include how many consumers completed the PHQ-9, symptom remission rates, and how many consumers are making progress toward feeling better (defined as at least 50 percent reduction in the baseline PHQ-9 score). These measures were incorporated into the work of Minnesota Community Measurement, which maintains a website that publicly reports these measures for clinics participating in DIAMOND and other primary care and behavioral health clinics across the state (http://www.mnhealthscores.org). The website facilitates comparisons of clinics over time and reports state averages. The Minnesota measures are endorsed by NQF and included in the CMS EHR Incentive programs. Other delivery systems (Kaiser Permanente and Group Health, for example) and community initiatives (MaineHealth, for example) are adopting these measures and similar measurement and reporting systems.

Other health plan initiatives have also used repeated outcome assessments that give feedback to providers and/or consumers. For example, Optum Behavioral Health, a large national managed behavioral health organization with more than 100,000 providers in its network, uses the Algorithms for Effective Reporting and Treatment (ALERT) system. This system combines data from a consumer-reported Wellness Assessment with claims data to track consumer improvement and identify individuals who are at risk for poor outcomes. The ALERT system identifies consumers with "high distress" or who are at risk of substance abuse who demonstrate poor progress early in treatment. The one-page Wellness Assessment contains items derived from validated tools that assess symptom severity, functional impairment, self-efficacy, substance abuse risk, and the presence of co-morbid medical conditions. The provider administers the assessment when treatment begins and then again during later visits. With permission from the consumer, Optum mails a follow-up assessment four months after treatment begins.

Several web-based systems for tracking symptoms and functioning have been used for quality monitoring and improvement in mental health care settings. One example is the Treatment Outcome Package (TOP), which tracks mental health symptoms and functioning across 12 clinical domains (Kraus 2012). Providers use this system to email a link to the consumer to complete an online questionnaire (which requires 3-5 minutes). The system scores the questionnaire and generates a short report for the provider. Over time, these reports graphically display changes in scores within each domain and benchmark those scores to the general non-clinical population. The report alerts the provider if the consumer is not making progress as expected and includes a list of suggested treatment practices aimed at improving outcomes. TOP also generates a section of the report designed to give to the consumer as feedback. Providers also receive monthly aggregate reports that benchmark their risk adjusted performance against similar professionals. Health care systems have used this system widely. For example, Blue Cross and Blue Shield of Massachusetts incentivized the use of TOP by requiring that providers achieve certain response rates on the tool in order to receive their annual provider fee increase. This was not without controversy and pushback from providers, but ultimately, the TOP was administered over 40,000 times in the first six months of the program (Youn et al. 2012; Liptzin 2009; Blais et al. 2009).

Another example of an approach to outcomes monitoring is the Partners for Change Outcome Management System (PCOMS) International Center for Clinical Excellence, which was recently listed in the SAMHSA National Registry of Evidence-based Programs and Practices (Reese et al. 2010; Anker et al. 2009; Campbell and Hemsley 2009). PCOMS consists of two brief scales: (1) the Outcome Rating Scale (ORS), which assesses mental health functioning and distress and the consumer's perceived benefit of treatment; and (2) the Session Rating Scale (SRS), which assesses the consumer's perception of the therapeutic alliance. The provider administers the ORS at the beginning of the therapy session and the SRS toward the end of the session. The provider and consumer discuss the consumer's ratings for both measures on a session-by-session basis to encourage the consumer's engagement in treatment, improve therapeutic alliance, and keep the sessions focused on the concerns identified by the consumer.

A final example of routine outcomes monitoring is the Improving Access to Psychological Therapy program, which currently operates throughout much of England (Department of Health 2012). This treatment model contains several components, including the use of assessments that identify the individual's concerns and treatment goals at initial contact, and tracks symptom reduction and progress toward treatment goals. Participating providers must ensure that at least 90 percent of consumers who are seen at least twice receive pre-treatment and post-treatment assessments and have a score on the main outcome measures. In addition, these providers receive weekly feedback and clinical supervision to discuss adjusting treatment based on information from the assessments. Information from these assessments, as well as other clinical information, is stored in an electronic database that therapists and care managers can access to monitor consumer progress and managers can use to monitor care and identify opportunities for quality improvement.

The measurement strategies described above, as well as others, have demonstrated promising results. Several randomized trials have found that assessing symptoms and functioning at regular intervals and giving the results as feedback to providers helps to identify individuals at risk for poor outcomes, prevents the worsening of symptoms, and decreases the time to positive outcomes (Bickman et al. 2011; Whipple and Lambert 2011; Shimokawa et al. 2010; Lambert et al. 2005). One meta-analysis of trials that examined feedback given to mental health providers during the course of treatment found a modest positive effect on short-term consumer outcomes but no effect on treatment duration, costs, or longer-term consumer outcomes (although very few studies included information on treatment costs or duration) (Knaup et al. 2009). The same meta-analysis found that feedback to providers had a larger positive effect on short-term outcomes if: (1) the feedback included information on mental health progress over time (versus providing information about current status only); (2) both the consumer and provider received feedback (versus only one of them); and (3) feedback was given more than once. These findings are consistent with the experience of one managed behavioral health organization, which found that six-month outcomes were better among consumers whose therapist reported using the information provided in the progress reports compared with consumers whose provider received the reports but did not report using them (Azocar et al. 2007). These findings underscore the importance of having user-friendly mechanisms that enable providers and consumers to use the feedback from outcome measures.

A. Advantages

Outcome measures can be used for clinical decision making and to engage consumers in care. As described in the examples above, routine outcomes monitoring can serve at least two purposes: (1) to help track consumer progress and identify individuals who fail to respond to treatment; and (2) to encourage consumer engagement in treatment.

Identifying individuals who fail to respond to treatment is particularly important. One study of over 6,000 individuals who received an average of four psychotherapy sessions in community-based treatment settings found that only about 35 percent improved and about 8 percent experienced worse symptoms or functioning (Hansen et al. 2002). Likewise, studies of children and youth receiving mental health care have found that as many as 24 percent get substantially worse during treatment (Warren et al. 2010). By drawing on data from outcome assessments, the measurement systems described in this paper are able to identify individuals experiencing treatment failure by comparing the trajectory of their progress with statistically generated expected results (Boswell et al. 2013). Some studies have suggested that such measurement approaches can identify 85-100 percent of individuals who get worse during treatment (Ellsworth et al. 2006; Lambert et al. 2002), which is better than relying on clinical judgment alone (Hannan et al. 2005). The information from assessments can also be combined with claims data to implement algorithms that identify individuals at risk for hospitalization (McAleavey et al. 2012).

Consumers can directly benefit from receiving feedback on outcome measures. Some clinicians and researchers have noted that even slight improvements in outcome measures can be encouraging for consumers and can help enhance therapeutic alliance (Youn et al. 2012). Some routine outcome monitoring systems include mechanisms to collect additional information from consumers at risk for poor outcomes. For example, a consumer who is not improving would be asked to report additional information about his or her social support and recent life events, which may provide critical contextual information to help the therapist adjust treatment (Boswell et al. 2013). Thus, repeated assessments may provide another tool for consumers to share information, engage in discussion about their treatment goals and progress, and make treatment decisions.

Outcome measures may overcome the limitations of structure and process measures. To some extent, the measurement of outcomes can be accomplished without regard to the specific content of the treatment being delivered. Outcome measurement shifts the focus from the content of the treatment, which may vary across consumers and time, to the result of the treatment. Nonetheless, measuring how psychotherapy was delivered in tandem with outcomes may help to provide insight into which components of therapy and common factors across psychotherapies produce positive results in typical care settings. Measures that focus directly on outcomes and inform clinical decision making as well as quality improvement may be more appealing than measures of specific processes of care. There may not be agreement on what constitutes high quality treatment, or when there is inadequate evidence that the specific structures and processes of care are strongly associated with outcomes.

In sum, outcome measures may have benefits at multiple levels of the health care system. The measurement of outcomes offers an approach for making care more patient-centered by enabling consumers (as well as other important individuals in their lives) to report information about their symptoms and functioning. For consumers, measures that focus on functional outcomes--such as relationships, employment, and engagement in the community--provide direct feedback that they can use for self-monitoring and making treatment decisions. Providers can use such measures to identify individuals who are not responding to treatment or may require adjustments to their treatment. Likewise, improvements in consumer outcomes may signal to consumers, providers, or health plans that more intensive or sustained treatment may have limited benefit to the individual or family. The tools used to monitor the progress of treatment can be adapted to the service setting and the population. Providers, health plans, and the broader health care system can use performance on these measures to monitor outcomes and identify opportunities for quality improvement as well as promising practices.

B. Challenges

Health systems must overcome several obstacles to widely implement psychotherapy outcome measures, including: (1) selecting outcome measures that are meaningful for consumers, providers, and other stakeholders; (2) deciding on the appropriate level of reporting and strategies for making fair comparisons across providers, plans, or systems; (3) overcoming providers' lack of familiarity with outcome measures; and (4) incorporating the administration and reporting of measures into clinical workflows so that it becomes a routine part of therapy (Boswell et al. 2013; Harding et al. 2011). Here we briefly discuss these challenges.

Selecting appropriate outcome assessments. There are many choices of outcome assessments. Some assessments focus on symptom severity among specific diagnostic groups; others focus more broadly on functioning across various life domains. Some of these assessments are proprietary. The selection of the best assessment may depend on the target population, treatment setting, and end use of the information. Although it is important to have some flexibility in which instruments are used to assess outcomes, the use of very different assessments across providers, health plans, or states may impede comparisons. In addition, because providers typically belong to several health plans and receive reimbursement through various state and federal funding streams--each with their own reporting requirements--aligning outcome reporting efforts would decrease burden.

In addition to measuring symptoms and functioning, experts and measurement stakeholders, including the Measures Application Partnership Dual Eligible Beneficiaries Workgroup, have recommended that performance measures should assess goal setting and goal achievement (NQF 2012). For example, the framework guiding measure-development for the CMS EHR incentive program calls for steps in building performance measures based on patient-reported outcomes. Experts recommended gradually introducing performance measures that include goal setting, goal attainment, and improvement in outcomes over time (Torda 2013). This goal setting approach is drawing on literature on goal attainment scaling developed in the mental health field and implemented in a variety of settings and populations (Kiresuk et al. 1994).

Ideally, outcome assessments would be inexpensive to collect, impose the least amount of data collection burden to providers and consumers, apply to broad populations, and have some comparability in terms of reliability and validity. Stakeholders could draw from a common menu of outcome assessments that would facilitate comparisons. There are some resources on which to build; the MacArthur Foundation and Project IMPACT have undertaken efforts to develop a tool kit of measures for depression (Harding et al. 2011), and the Patient-Reported Outcomes Measurement Information System initiative, sponsored by the HHS National Institutes of Health, has assembled measures and items that assess symptoms and functioning. When these measures are used for quality improvement within an organization, the goal may be simply improvement without a specific target. However, if different measures were used across providers or health plans for the purposes of public reporting and accountability, health plans or other entities would need to use methods to make equitable comparisons. Statistical concepts, such as the Reliable Change Index, could be used.

Lack of familiarity with outcomes monitoring. Historically, academic training for psychiatrists and other behavioral health providers has not included learning how to incorporate outcome assessments into clinical practice. Some providers may be reluctant to use assessments because they fear it could damage their relationships with consumers or threaten their autonomy (Boswell et al. 2013). Providers may also not see the value of using repeated assessments based on standardized scales (Lambert et al. 2005) or may not want their outcomes compared with other therapists (Youn et al. 2012; Okiishi et al. 2006).

Providers would need training to learn how to introduce assessments to consumers, interpret the results, and use those results for clinical decision making and quality improvement. There is surprisingly little research to guide how providers should approach the steps involved in introducing and administering mental health assessment tools and discussing the results of those tools with consumers (Wissow et al. 2013).

Getting providers to use feedback can be a formidable challenge. One study found that even when outcome assessments were required, most mental health providers reported that they did not use the feedback for clinical decision making (Garland et al. 2003). It is unclear what strategies work best for providing feedback and how to structure that feedback in a manner that is helpful. Most outcome measurement and feedback strategies in psychotherapy do not give specific instruction on how providers should use feedback, allowing providers to use their clinical judgment instead (Lambert et al. 2005). Some feedback strategies have successfully used color-coded systems that correspond to consumer progress, and there is some evidence that certain clinical support tools can help providers use feedback, but further research is needed to understand how these strategies and tools can work in typical community-based treatment settings (Whipple and Lambert 2011). Health plans have employed various strategies for giving feedback to providers. For example, PacifiCare Behavioral Health sent quarterly reports to providers that contained a summary of progress for their patient population (Brown and Jones 2005). They also sent letters to providers when a consumer failed to demonstrate improvement, which encouraged the provider to keep that individual engaged in treatment and offered to pre-authorize more intensive services. They also sent letters to providers when individuals responded well to treatment; the idea was to acknowledge the good outcome and suggest that longer-term treatment might not necessarily result in a better outcome. It is unclear to what extent providers may perceive such feedback as limiting their autonomy or attempting to restrict access to care--potentially adding to their reservations for participating in outcome measurement systems.

Making outcome measurement part of routine care. Providers and health plans will need to adopt new processes for measuring consumer outcomes and for using this information to improve the quality of care. Some providers may not have the time or resources to administer, score, or interpret assessments. Reimbursement models that pay for the administration of such outcome measures in mental health, similar to routine medical tests, may encourage their use (Boswell et al. 2013). There may also be opportunities to integrate these measures into existing reporting programs, such as Meaningful Use, which would provide a financial incentive. Moreover, providers will be challenged to incorporate the use of measures into their routine practice as opposed to an additional activity. Some providers with limited time may be able to rely on non-clinical staff, such as medical assistants or care managers, to administer and score the measures. Web-based or computer-based screening tools could ease the administration of measures, but smaller practices might not find investments in these technologies feasible. Health plans or large health care delivery systems may be well-positioned to provide the infrastructure to facilitate the measurement of outcomes and give feedback to providers, as in the examples described above.

Making fair comparisons. If outcome measures are used for public reporting or accountability they may require risk adjustment for two reasons: (1) to ensure that performance is not attributable to differences in severity of illness or other factors that may be beyond the control of the provider, organization, health plan, etc.; and (2) to guard against the possibility that providers or health plans would have an incentive to avoid treating/enrolling consumers with more severe problems. Methods for risk adjustment in mental health care are limited, in part because of the incomplete data on severity of illness and other consumer characteristics in claims or medical records. Some of the outcome measurement systems described in this paper have employed risk adjustment strategies that may offer guidance, whereas others are proprietary. Some potential alternatives to risk adjustment include reporting on stratified populations or examining whether there was meaningful change in clinical care in response to lack of improvement (Kerr et al. 2012). In addition, states, health plans, and providers could use measures to monitor incremental improvements rather than absolute values (Kilbourne et al. 2010). For example, a provider or health plan would be held accountable for whether consumer outcomes are improving or meeting a benchmark from one year to the next rather than being assessed and compared at only one point in time. Another possible variation might be to provide incentives to organizations that can demonstrate clinical improvement for a minimally acceptable percentage of their consumers rather than require improvement across the entire cohort.

VII. Conclusions and Next Steps

Measures that assess the content, duration, and outcomes of psychotherapy have the potential to inform quality improvement efforts and bring greater accountability to the delivery of mental health care. Several different measurement strategies could offer insight into the quality of psychotherapy; all would require investments, varying levels of changes in workflow for providers, and consumer participation.

Decisions about the development and implementation of quality measures for psychotherapy requires the collaboration of many stakeholders, including consumers, providers, health plans, payers, and state agencies. In light of the strengths and limitations of the different measurement strategies described in this paper, these stakeholders may need to prioritize measure-development and implementation. In this final chapter, we identify some immediate and longer-term opportunities for psychotherapy quality measures within the context of existing data sources, reporting structures, and ongoing health care reforms. We primarily focus on measures assessing the effectiveness and outcomes of care rather than measures that assess the utilization or costs of services.

A. Opportunities for Psychotherapy Measures

Based on our synthesis of the strengths and limitations of measurement approaches and feedback from our expert panel, we propose prioritizing the measurement of outcomes. This would involve developing the infrastructure for outcomes monitoring and building capacity for delivering evidence-based psychotherapies. Outcome measurement efforts must ensure consumer participation in guiding measure-development, allow flexibility in the choice of measures, protect the privacy and confidentiality of outcomes data, and include incentives and support for providers who make these important but sometimes radical changes in practice. Structure measures may help to facilitate the development of the infrastructure to measure outcomes and ensure that providers have the ability to offer evidence-based treatments. Tracking the use of psychotherapy will be a useful adjunct for these efforts. In the longer term, efforts should focus on further operationalizing and reporting outcome measures and building process measures that will help support quality improvement.

In other areas of health care, process measures have preceded the development and implementation of outcome measures, in part because a strong evidence base and consensus among stakeholders supported the identification of processes expected to lead to positive outcomes (for example, glucose testing for diabetes) and because these processes were relatively straightforward to measure (for example, did the patient receive the glucose test?). Defining these processes of care for psychotherapy is much more complicated and controversial. Although certain types of psychotherapy have strong evidence, there is not widespread agreement in the literature or among clinicians on which specific processes of care or treatment elements lead to positive outcomes. For example, there is not clear evidence that a psychotherapy session that involves assigning homework yields superior outcomes compared with a session that does not (Bell et al. 2013). In addition, the delivery of psychotherapy may be adapted to fit into the context of the service environment and meet the needs of the consumer, but it is unclear to what extent these adaptations in the processes of care influence outcomes. Bickman (2008) summarized that evidence-based treatments in mental health "are not structured in a way that they can be mechanically implemented without variations introduced by the clinician and the service organization. Mental health services will not be successful in removing the influence of the clinician or 'clinician-proofing' treatments any more than the field of education has been successful in 'teacher-proofing' the curriculum." Such adaptations, as well as the lack of research to understand how they influence outcomes, present challenges for developing structure and process measures that can be used for accountability. Focusing on steps that lead to the measurement of outcomes may offer the most value to the broadest group of stakeholders. A focus on outcomes would be consistent with changes underway to measures in medical care, where new guidelines are calling into question some established measures, including those for cardiovascular disease and breast cancer.

As there are limitations with any single measurement approach or data source, it is likely that in the long-term different types of measures that draw on various data sources will be necessary to fully understand how psychotherapy is delivered and whether it yields positive outcomes. Using multiple measures could generate information that helps to identify the relationship between structure, process, and outcomes in psychotherapy. For example, information from structure measures, such as whether providers have certain types of training and use specific procedures, could be coupled with data from EHRs and outcome assessments to elucidate the link between the structures that support psychotherapy, the delivery of specific psychotherapeutic content, and the outcomes of treatment. Measuring outcomes while simultaneously trying to understand the structures and processes of care may help to identify the most effective forms of psychotherapy and build the evidence for those psychotherapies with less empirical support (Garland et al. 2010a).

1. Immediate Opportunities

Implement structure measures that build capacity for delivering evidence-based psychotherapies and lead to the measurement of outcomes. As described above, there are several examples of practical tools and systems that have been used to assess the outcomes of psychotherapy. Several of these tools and measurement systems have a substantial literature supporting their effectiveness in improving outcomes and use in large-scale quality improvement initiatives; in some ways, their use has become an evidence-based practice. Some of these tools and systems have been used primarily for psychotherapy research or clinical training, and others have been implemented by large provider networks and health plans. The basic infrastructure to support the measurement of psychotherapy outcomes--including such things as patient registries and electronic data collection and feedback systems--is available to some providers via health plans or delivery systems and could serve as a model. Nonetheless, the widespread adoption of routine outcomes monitoring for psychotherapy would require further investments in the data collection and reporting infrastructure.

Structure measures that assess the capacity to deliver psychotherapy could draw upon existing methods for assessing fidelity in research settings and standards for other aspects of evidence-based practices that are incorporated in requirements for health homes. These measures could address training and supervision of staff as well as whether providers/entities have protocols and systems in place to conduct routine consumer assessments, match treatment to consumer needs and preferences, and adapt treatment when consumers do not improve on outcome measures.

States could position the structure measures within the certification requirements of their Medicaid and mental health agencies, health plans, and ACOs for provider networks. These structure measures could also be used within health homes, patient-centered medical homes, or other certification programs. These measures would ideally focus on mental health clinics, primary care practices, or other provider organizations rather than individual clinicians or therapists. They would focus on whether the organization has staff with basic training and credentials in evidence-based psychotherapy, protocols for supervision and stepped care, and the infrastructure to measure outcomes and use outcomes data to inform clinical decision making and quality improvement. States and health plans already have some basic structure measures in place to determine whether individual clinicians meet licensing requirements, and some states impose additional requirements for mental health providers to bill for specific types of services. States that require formal credentialing in specific evidence-based treatments and health plans that offer outcomes monitoring infrastructure may serve as examples for future efforts.

Offer outcome measurement approaches that are flexible but allow for comparisons. Many tools could be used to measure outcomes; each has strengths and limitations, and applications to various populations and settings. Health plans, clinics, and providers may benefit from selecting among tools that are applicable to the population and community they serve and that can be used in the context of their particular service setting. As mentioned above, there are some repositories of tools that could be useful. Consistent with the principles of other patient-reported outcome measures, the tool selected must be reliable and valid; "person centered"; meaningful for consumers, their families, and providers; and amenable to change (NQF 2013). Moreover, they must be easy to administer and interpret, they must yield information that providers and consumers can use to engage in the discussion of treatment goals, and they must foster progress toward reaching those goals. Finally, the systems that support or collect these measures must offer different modes to collecting the data (web-based, mail, telephone) and present the data back to providers and consumers in a manner that is useful for making treatment decisions and informing quality improvement. Some of the examples described in this paper may serve as models.

Offer incentives, training, and ongoing support. We should not underestimate the extent to which the routine measurement of outcomes and the use of outcomes data to guide clinical decision making and monitor quality introduce a major paradigm shift for many providers. Some providers are not accustomed to using standardized tools or assessments to monitor progress, and they may feel uncomfortable reporting information to other entities. It would be naïve to think that providers will change practice only because there is literature supporting the benefits of routine outcomes monitoring (Boswell et al. 2013).

Although payers and health plans have a financial incentive to ensure that providers monitor and obtain positive outcomes in an effort to minimize the use of more costly services, in the current reimbursement environment, measuring outcomes (or collecting process measures) would impose burden on providers--particularly small or solo practices that are unaffiliated with larger health care delivery systems. Many of these providers do not have the staff or other resources to complete and submit routine measures. Within the flexibility of Medicaid health homes and managed care arrangements, states and health plans could offer incentives for collecting and reporting psychotherapy outcomes and for achieving desirable outcomes (Bao et al. 2013). These incentives could come in the form of actual payment for the administration and reporting of outcome measures or by increasing referrals to high-performing providers (Boswell et al. 2013). Providers could also receive incentives for adopting some of the structures that may support outcomes monitoring, such as adoption of clinical registries and systems for collecting data or by using existing systems to demonstrate significant patient improvement.

Moreover, providers who are accustomed to relying solely on their clinical judgment will need training and support to understand how to integrate the results of routine measures into their decisions about how to tailor individual treatments and how to improve the effectiveness of the care for all consumers (Boswell et al. 2013). Such training could take the form of continuing education programs or training in graduate programs. Following the examples described in this paper, health plans may be well-positioned to offer decision support by giving feedback to providers in a manner that offers guidance on best practices and gives providers an opportunity to reflect on difficult cases.

Ensure confidentiality, security, and appropriate use of data. Providers, consumers, and their families need assurances that the sensitive information they provide is confidential and secure and will be used for the specific purposes. State agencies, health plans, or other entities must also be transparent in what information will be publically reported.

Individual providers may be reluctant to have their performance publically reported or they may have an insufficient number of consumers to report on. Attributing outcomes to an individual provider is also questionable, given that individuals often receive care from several providers and the very real possibility that factors outside of the provider's control account for outcomes. The challenge of attributing outcomes to a provider might argue for measuring outcomes at the patient level for clinical decision making but publicly reporting only at the health plan or clinic/organization level, which would align with other efforts to measure system performance (Berenson et al. 2013; Conway et al. 2013). Such an approach could also encourage a more collaborative effort focused on quality improvement at the organizational level.

Engage consumers and other stakeholders in the development of outcome measurement systems. As state agencies, health plans, and other entities engage in developing and implementing measurement strategies, they must ensure that the measures are consistent with the goals, values, and cultural diversity of consumers and their families, as well as other stakeholders. Consumers and family members must be engaged in selecting salient measures and developing processes that provide usable feedback from those measures. In clinical practice, consumers and their family members will benefit from having a choice of measures that cover domains of functioning that represent meaningful progress toward reaching goals rather than measures that only assess the remediation of symptoms or satisfaction with care. Likewise, payers, health plans, state and county agencies, and providers must provide input into the selection of measures.

The outcomes of psychotherapy are also relevant to stakeholder groups that may be somewhat outside of mainstream health or behavioral health care, such as education and juvenile justice systems or other state agencies. It is important to consider how these measures could be informative for these other groups. For instance, education systems may be interested in which provider organizations achieve outcomes that are meaningful for school systems, such as absenteeism.

Track the use of psychotherapy. Given that many individuals with mental health problems do not receive psychotherapy (Brown et al. 2012), there may be value in using claims data to track the use of services. As described above, currently available claims data, and Medicaid data in particular, lack information on the content of the therapy provided, and there are limitations in using this data to make comparisons across providers, health plans, or states. Nonetheless, claims data could be used to understand whether individuals with an indicated condition or medication have any contact with the mental health care system and maintain that contact. Such measures may not yield information that is necessarily indicative of the quality of psychotherapy, but they could point to patterns of use that merit further investigation.

2. Longer-Term Opportunities

In this section we describe some of the activities that could facilitate refining and reporting outcome measures and improve the data sources that would facilitate process measures that support quality improvement.

Clarify the conceptualization of outcomes and demonstrate methods for reporting outcomes measures. Although there are many examples of outcome measurement approaches, work remains to further refine the methods for using outcome measurement strategies for accountability. Measures that are primarily used for clinical supervision or training may not be appropriate for the purposes of accountability or large-scale quality improvement. As described in the examples above, the outcomes in many of these measurement systems are conceptualized quite differently. Some focus on remission of symptoms and functioning within specific domains of mental health, such as depression, but others are much broader. As measurement systems are implemented and data accumulates, it will be important to understand how different conceptualizations and strategies for measuring outcomes function among different populations and in different contexts. With more data, systems may be more equipped to implement measures that are applicable to broad diagnostic and demographic groups and can be used across settings.

Given the many different ways in which outcomes could be conceptualized and measured, there may be opportunities and particular value in building on ongoing efforts to design measures around goal attainment, which is consumer-directed. More research is needed to understand how goal attainment scales should be calculated for the purposes of accountability. Further, there is a need for measurement experts to refine risk adjustment strategies and look for alternatives to risk adjustment to hold entities accountable for care and facilitate fair comparisons.

Enhance billing codes to facilitate the use of claims data for quality measurement. In the future, measures that capture the content and duration of psychotherapy could theoretically use claims data if there were major improvements in the specificity of billing codes and if those codes were tied to some type of credentialing or linked to outcomes. CMS, state Medicaid programs, and health plans could develop more descriptive billing codes that correspond to specific psychotherapeutic processes, and they could restrict the use of such codes to providers who demonstrate competency or have certain credentials (as is the case with some other mental health billing codes, such as Multisystemtic Therapy or Assertive Community Treatment). Again, these processes are currently not well-defined for psychotherapy, and it would take considerable effort to achieve consensus among professional societies, payers, health plans, and state agencies on which processes of care are linked to outcomes and should therefore receive their own billing codes, and possibly differential payment. In addition, given that providers typically belong to many different health plans and receive reimbursement through various streams, it is a considerable investment for state Medicaid programs or health plans to create new codes and encourage their use. The use of new coding schemes would also impose an administrative and time burden on providers, especially if those schemes varied across health plans or other payers. Even with some standardization of codes, it is likely that their use will always vary somewhat across providers, states, and health plans, given the differences in how providers are reimbursed at the state and local levels. This might limit comparisons.

Incorporate fields on the content of psychotherapy in EHRs. EHRs could incorporate data elements that capture delivery of specific psychotherapeutic content. The VA is beginning to capture some specific elements of psychotherapy in its EHRs, which may yield findings about the feasibility of this measurement approach (Kilbourne et al. 2010). Measures that require documentation from EHRs would not currently capture information from the majority of behavioral health providers because most of them have not adopted EHRs, especially providers in small or solo practices who are not affiliated with a larger health care system. Measures that rely on EHRs to capture specific treatment elements would require providers to change their documentation practices--potentially creating additional documentation burden while becoming a "checkbox" measure. An alternative approach to having standardized fields in EHRs would be to develop more sophisticated computerized methods for extracting psychotherapeutic content from the notes of medical records. This approach has been tested and has produced some promising results (Shiner et al. 2013). However, much more work is needed for it to be applied to measures used for quality improvement and accountability. Moreover, the current reporting programs that use information from EHRs rely on structured fields.

Incorporate reporting on the content of psychotherapy into consumer surveys. Measures that examine the content of psychotherapy could use data reported directly from consumers. Health plans routinely conduct surveys to assess consumers' experiences with care, and they could also attempt to have consumers report on whether they received specific psychotherapeutic content. Although several widely used surveys assess consumers' experience with mental health treatment, they assess care in a more global fashion and do not attempt to measure whether the individual received specific psychotherapeutic content (Eisen et al. 2001). The work of Miranda and colleagues (2010) is a promising first step toward developing measures that attempt to elicit the psychotherapeutic content of visits directly from the consumer. More extensive testing of these measures would be needed to examine not only the extent to which consumers can accurately recall the content of psychotherapy but also how the reports are linked to outcomes.

Expand research on the effective ingredients of psychotherapy, with particular focus on specific consumer populations and conditions. Several gaps in research impede the development of quality measures for psychotherapy. There is a particular need to further identify the "active" ingredients as well as the common factors and therapeutic treatment processes that are associated with improvements in outcomes across different types of psychotherapy. Such research would help to inform the development of process measures that could be used across multiple types of therapy. As noted above, there is a need to better understand the effectiveness of various forms of psychotherapy among consumers with varying mental health needs and cultural backgrounds.

New measures would require testing to understand the best scoring method. In particular, there are no clear guidelines or standards for scoring the structure and process measures discussed in this paper. Some of the measures could be scored as an all-or-nothing measure. Alternatively, measures that assess whether providers deliver therapy as intended could be scored using some minimal threshold, as has been done in some studies (Barber et al. 2007). As researchers have noted, the thresholds vary across studies, and little information exists on the rationale behind the selection of the threshold. When developing a scoring algorithm, it is important to consider how the score will be used and the potential implications of high or low scores. For measures that use information from medical records/EHRs or surveys, the extent to which the delivery of specific content in a single therapy session is associated with consumer outcomes is unclear. It is conceivable that a provider could have one bad session and deliver high quality care in all other sessions so that, taken together, their sessions result in positive outcomes. In high-stakes situations where the score has significant implications, it might be important to consider using an overall score that presents a fair and accurate assessment of the quality of care provided. It might also make sense to score some types of measures only for an organization or health plan, not for individual providers.

B. Conclusion

Developing and implementing quality measures for psychotherapy requires a comprehensive plan to identify the end uses of measures, the data available to support measures, and measurement priorities. With this paper, we attempt to provide a foundation for developing such a plan by identifying measure options and describing their strengths and limitations. Focusing on steps that lead to the measurement of outcomes may offer the most value to the broadest group of stakeholders.

References

Aarons, G.A., C. Glisson, P.D. Green, K. Hoagwood, K.J. Kelleher, J.A. Landsverk, Research Network on Youth Mental Health, J.R. Weisz, B. Chorpita, R. Gibbons, C. Glisson, E.P. Green, K. Hoagwood, P.S. Jensen, K. Kelleher, J. Landsverk, S. Mayberg, J. Miranda, L. Palinkas, and S. Schoenwald. "The Organizational Social Context of Mental Health Services and Clinician Attitudes Toward Evidence-Based Practice: A United States National Study." Implementation Science, vol. 7, 2012, p. 56.

Aarons, G.A., M. Hurlburt, and S. M. Horwitz. "Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors."Administration and Policy in Mental Health, vol. 38, no. 1, 2011, pp. 4-23.

Aarons, G.A., R.S. Wells, K. Zagursky, D.L. Fettes, and L.A. Palinkas. "Implementing Evidence-Based Practice in Community Mental Health Agencies: A Multiple Stakeholder Analysis." American Journal of Public Health, vol. 99, no. 11, 2009, pp. 2087-2095.

Abbass, A., and E. Driessen. "The Efficacy of Short-Term Psychodynamic Psychotherapy for Depression: A Summary of Recent Findings." Acta Psychiatrica Scandinavica, vol. 121, no. 5, 2010, pp. 398.

Adair, C.E., G.M. McDougall, C.R. Mitton, A.S. Joyce, T.C. Wild, A. Gordon, N. Costigan, L. Kowalsky, G. Pasmeny, and A. Beckie. "Continuity of Care and Health Outcomes Among Persons with Severe Mental Illness." Psychiatric Services, vol. 56, no. 9, 2005, pp. 1061-1069.

Agency for Healthcare Research and Quality (AHRQ). "Selecting Structure Measures for Clinical Quality Measurement." Available at http://www.qualitymeasures.ahrq.gov/tutorial/Structure Measure.aspx. Accessed November 9, 2013a.

Agency for Healthcare Research and Quality. "Initiative Features Fixed Monthly Payments to Primary Care Clinics for Providing Depression Care Bundle, Allowing Many Patients to Achieve Good Outcomes." Available at http://www.innovations.ahrq.gov/content.aspx?id=3838. Accessed November 11, 2013b.

Ahn, H., and B.E. Wampold. "Where Oh Where Are the Specific Ingredients? A Meta-Analysis of Component Studies in Counseling and Psychotherapy." Journal of Counseling Psychology, vol. 48, no. 3, 2001, pp. 251-257.

American Academy of Child and Adolescent Psychiatry (AACAP). "Practice Parameter for Cultural Competence in Child and Adolescent Psychiatric Practice." April 16, 2013. Available at http://www.aacap.org/App_Themes/AACAP/docs/practice_parameters/Cultural_Competence_Web.pdf. Accessed February 15, 2014.

American Psychiatric Association. "Practice Guidelines for the Treatment of Major Depression." Washington, DC: American Psychiatric Press, Inc., 2006.

American Psychological Association. "2013 Psychotherapy CPT Codes for Psychologists." http://www.apapracticecentral.org/reimbursement/billing/psychotherapy-codes.pdf. Accessed November 10, 2013.

Anastario M.P., H.P. Rodriguez, P.M. Gallagher, P.D. Cleary, D. Shaller, W.H. Rogers, K. Bogen, and D.G. Safran. "A Randomized Trial Comparing Mail Versus In-Office Distribution of the CAHPS Clinician and Group Survey." Health Services Research, vol. 45, 2010, pp.1345-1359.

Anker, M.G., B.L. Duncan, and J.A. Sparks. "Using Client Feedback to Improve Couple Therapy Outcomes: A Randomized Clinical Trial in a Naturalistic Setting."Journal of Consulting and Clinical Psychology, vol. 77, no. 4, 2009, pp. 693-704. Available at http://www.ncbi.nlm.nih.gov/pubmed/19634962.

Arean, P.A., P.J. Raue, J.A. Sirey, and M. Snowden. "Implementing Evidence-Based Psychotherapies in Settings Serving Older Adults: Challenges and Solutions."Psychiatric Services, vol. 63, no. 6, 2012, pp. 605-607.

Australian Centre for Posttraumatic Mental Health (ACPMH). "Australian Guidelines for the Treatment of Adults with Acute Stress Disorder and Posttraumatic Stress Disorder." Melbourne, Victoria: ACPMH, 2007.

Azocar, F., B. Cuffel, J. McCulloch, J.F. McCabe, S. Tani, and B.B. Brodey. "Monitoring Patient Improvement and Treatment Outcomes in Managed Behavioral Health." Journal for Healthcare Quality, vol. 29, no. 2, 2007, pp. 4-12.

Baldwin, S.A., A. Berkeljon, D.C. Atkins, J.A. Olsen, and S.L. Nielsen. "Rates of Change in Naturalistic Psychotherapy: Contrasting Dose-Effect and Good-Enough Level Models of Change." Journal of Consulting and Clinical Psychology, vol. 77, no. 2, 2009, pp. 203-211.

Bao, Y., L.P. Casalino, and H.A. Pincus. "Behavioral Health and Health Care Reform Models: Patient-Centered Medical Home, Health Home, and Accountable Care Organization." Journal of Behavioral Health Services and Research, vol. 40, no. 1, 2013, pp. 121-132.

Barber, J.P., E. Triffleman, and C. Marmar. "Considerations in Treatment Integrity: Implications and Recommendations for PTSD Research." Journal of Traumatic Stress, vol. 20, no. 5, 2007, pp. 793-805.

Baumann, B.L., D.J. Kolko, K. Collins, and A.D. Herschell. "Understanding Practitioners' Characteristics and Perspectives Prior to the Dissemination of an Evidence-Based Intervention." Child Abuse and Neglect, vol. 30, no. 7, 2006, pp. 771-787.

Beck, J.S. and A.T. Beck. "Cognitive Behavior Therapy: Basics and Beyond." New York, NY: Guilford Publication, 2011.

Bell, E.C., D.K. Marcus, and J.K. Goodlad. "Are the Parts as Good as the Whole? A Meta-Analysis of Component Treatment Studies." Journal of Consulting and Clinical Psychology, vol. 81, no. 4, 2013, pp. 722-736.

Berenson, R.A., P.J. Pronovost, and H.M. Krumholz. "Achieving the Potential for Health Care Performance Measures." May 2013. Robert Wood Johnson Foundation and Urban Institute. Available at http://www.rwjf.org/content/dam/farm/reports/reports/2013/rwjf406195. Accessed July 12, 2013.

Bernal, G., M.I. Jiménez-Chafey, R. Domenech, and M. Melanie. "Cultural Adaptation of Treatments: A Resource for Considering Culture in Evidence-Based Practice." Professional Psychology: Research and Practice, vol. 40, no. 4, 2009, pp. 361-368.

Bickman, L., S.D. Kelley, C. Breda, A.R. de Andrade, and M. Riemer. "Effect of Routine Feedback to Clinicians on Mental Health Outcomes of Youths: Results of a Randomized Trial." Psychiatric Services, vol. 62, no. 12, 2011, pp. 1423-1429.

Bickman, L. "A Measurement Feedback System (MFS) Is Necessary to Improve Mental Health Outcomes." Journal of the American Academy of Child and Adolescent Psychiatry, vol. 47, no. 10, 2008, pp. 1114-1119.

Bickman, L. "A Common Factors Approach to Improving Mental Health Services." Mental Health Services Research, vol. 7, no. 1, 2005, pp. 1-4.

Blais, M., R.G. Frank, A.A. Nierenberg, and S.L. Rauch. "Measuring Outcomes Makes Good Sense." Psychiatric Services, vol. 60, 2009, p. 112.

Bond, G.R., D.R. Becker, and R.E. Drake. "Measurement of Fidelity of Implantation of Evidence-Based Practices: Case example of the IPS Fidelity Scale." Clinical Psychology: Science and Practice, vol. 18, 2011, pp. 125-140.

Bond, G.R., R.E. Drake, G.J. McHugo, A.E. Peterson, A.M. Jones, and J. Williams. "Long-Term Sustainability of Evidence-Based Practices in Community Mental Health Agencies." Administration and Policy in Mental Health, 2012, online ahead of print.

Boswell, J.F, D.R. Kraus, S.D., Miller, and M.J. Lambert. "Implementing Routine Outcome Monitoring in Clinical Practice: Benefits, Challenges, and Solutions."Psychotherapy Research, 2013, online ahead of print.

Brookman-Frazee, L., R.A. Haine, M. Baker-Ericzen, R. Zoffness, and A.F. Garland. "Factors Associated with Use of Evidence-Based Practice Strategies in Usual Care Youth Psychotherapy." Administration and Policy in Mental Health, vol. 37, no. 3, 2010, pp. 254-269.

Brookman-Frazee, L., R.A. Haine, E.N. Gabayan, and A.F. Garland. "Predicting Frequency of Treatment Visits in Community-Based Youth Psychotherapy."Psychological Services, vol. 5, no. 2, 2008, pp. 126-138.

Brown, J.D., A. Barrett, H.T. Ireys, E. Caffery, and K. Hourihan. "Evidence-Based Practices for Medicaid Beneficiaries with Schizophrenia and Bipolar Disorder." Report prepared for the U. S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. Washington, DC: Mathematica Policy Research, April 2012. Available at http://aspe.hhs.gov/daltcp/reports/2012/ebpsbd.shtml. Accessed July 12, 2013.

Brown, J.D., and S.H. Scholle. "Development and Testing of Behavioral Health Quality Measures: Recommendations for Areas of Measurement and Summary of Environmental Scan." Internal Project Memorandum prepared for the U. S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation, Contract Number HHSP23320095642WC, Task Order Number HHSP23320100019WI. Washington, DC: Mathematica Policy Research, January 20, 2012.

Brown, J.D., and L.S. Wissow. "Disagreement in Parent and Primary Care Provider Reports of Mental Health Counseling." Pediatrics, vol. 122, 2008, pp. 1204-1211.

Brown, J.D., L.S. Wissow, A. Gadomski, C. Zachary, E. Bartlett, and I. Horn. "Parent and Teacher Mental Health Ratings of Children Using Primary Care Services: Inter-Rater Agreement and Implications for Mental Health Screening." Ambulatory Pediatrics, vol. 6, 2007, pp. 347-351.

Brown, G.S., and E.R. Jones. "Implementation of a Feedback System in a Managed Care Environment: What Are Patients Teaching Us?" Journal of Clinical Psychology, vol. 61, no. 2, 2005, pp. 187-198.

Campbell, A., and S. Hemsley. "Outcome Rating Scale and Session Rating Scale in Psychological Practice: Clinical Utility of Ultra-Brief Measures." Clinical Psychologist, vol. 13, no. 1, 2009, pp. 1-9.

Campbell, B.K. "Fidelity in Public Health Clinical Trials: Considering Provider-Participant Relationship Factors in Community Treatment Settings." Journal of Public Health Dentistry, vol. 71, supp. 1, 2011, pp. S64-S65.

Cape, J., C. Whittington, M. Buszewicz, P. Wallace, and L. Underwood. "Brief Psychological Therapies for Anxiety and Depression in Primary Care: Meta-Analysis and Meta-Regression." BMC Medicine, vol. 8, no. 1, 2010, p. 38.

Chambless, D.L., and T.H. Ollendick. "Empirically Supported Psychological Interventions: Controversies and Evidence." Annual Review of Psychology, vol. 52, 2001, pp. 685-716.

Chapman, J.E., M.R. McCart, E.J. Letourneau, and A.J. Sheidow. "Comparison of Youth, Caregiver, Therapist, Trained, and Treatment Expert Raters of Therapist Adherence to a Substance Abuse Treatment Protocol." Journal of Consulting and Clinical Psychology, 2013, online ahead of print.

Chorpita, B.F., and E.L. Daleiden. "Mapping Evidence-Based Treatments for Children and Adolescents: Application of the Distillation and Matching Model to 615 Treatments from 322 Randomized Trials." Journal of Consulting and Clinical Psychology, vol. 77, no. 3, 2009, pp. 566-579.

Chorpita, B.F., and J. Regan. "Dissemination of Effective Mental Health Treatment Procedures: Maximizing the Return on a Significant Investment." Behaviour Research and Therapy, vol. 47, no. 11, 2009, pp. 990-993.

Centers for Medicare and Medicaid Services (CMS). "CMS Measures Inventory." Available at http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityMeasures/CMS-Measures-Inventory.html. Accessed April 28, 2014 (downloaded database file of measures dated March 31, 2014).

Centers for Medicare and Medicaid Services. "ICD-10-PCS Official Guidelines for Coding and Reporting." Available at http://www.cms.gov/Medicare/Coding/ICD10/Downloads/pcs_2013_guidelines.pdf. Accessed November 10, 2013.

Clark, M.S., K.L. Jansen, and J.A. Cloy. "Treatment of Childhood and Adolescent Depression." American Family Physician, vol. 86, no. 5, 2012, pp. 442.

Conway, P., H.F. Mostashari, and C. Clancy. "The Future of Quality Measurement for Improvement and Accountability." Journal of the American Medical Association, vol. 309, no. 21, 2013, pp. 2215-2216.

Cook, J.M., T. Biyanova, J. Elhai, P.P. Schnurr, and J.C. Coyne. "What Do Psychotherapists Really Do in Practice? An Internet Study of Over 2,000 Practitioners."Psychotherapy: Theory, Practice, Research, Training, vol. 47, no. 2, 2012, pp. 260-267.

Cuijpers, P., A.S. Geraedts, P. van Oppen, G. Andersson, J.C. Markowitz, and A. van Straten. "Interpersonal Psychotherapy for Depression: A Meta-Analysis."American Journal of Psychiatry, vol. 168, no. 6, 2011, pp. 581-592.

Czajkowski, S.M. "The Importance of Monitoring and Maximizing Treatment Fidelity in Public Health Research." Journal of Public Health Dentistry, vol. 71, supp. 1, 2011, pp. S67-S68.

Department of Health, National Health Service. "IAPT 3 Year Report: The First Million Patients. Available at http://www.iapt.nhs.uk/silo/files/iapt-3-year-report.pdf. Accessed February 20, 2014.

Department of Veteran Affairs and Department of Defense. "VA/DoD Clinical Practice Guidelines: Management of Post Traumatic Stress Disorder and Acute Stress Reaction." 2010. Available at http://www.healthquality.va.gov/Post_Traumatic_Stress_Disorder_PTSD.asp. Accessed June 17, 2013.

Donabedian, A. "The Quality of Care. How Can It Be Assessed?" Journal of the American Medical Association, vol. 260, no. 12, 1988, pp. 1743-1748.

Donabedian, A. "Exploration in Quality Assessment and Monitoring: the Definition of Quality and Approaches to its Assessment." Ann Arbor, MI: Health Administration Press, 1980.

Driessen, E., P. Cuijpers, S.C. de Maat, A.A. Abbass, F. de Jonghe, and J.J. Dekker. "The Efficacy of Short-Term Psychodynamic Psychotherapy for Depression: A Meta-Analysis." Clinical Psychology Review, vol. 30, no. 1, 2010, pp. 25-36.

Drapeau, M. "The Value of Tracking in Psychotherapy." Integrating Science and Practice, vol. 2, 2012, pp. 2-6.

Druss, B.G., and L. Dimitropoulos. "Advancing the Adoption, Integration and Testing of Technological Advancements within Existing Care Systems." General Hospital Psychiatry, vol. 35, no. 4, 2013, pp. 345-348.

Druss, B.G., P.S. Wang, N.A. Sampson, M. Olfson, H.A. Pincus, K.B. Wells, and R.C. Kessler. "Understanding Mental Health Treatment in Persons Without Mental Diagnoses: Results from the National Comorbidity Survey Replication." Archives of General Psychiatry, vol. 64, no. 10, 2007, pp. 1196-1203.

Eisen, S.V., J.A. Shaul, H.S. Leff, V. Stringfellow, B.R. Clarridge, and P.D. Cleary. "Toward a National Consumer Survey: Evaluation of the CABHS and MHSIP Instruments." Journal of Behavioral Health Services and Research, vol. 28, no. 3, 2001, pp. 347-369.

Ellsworth, J.R., M.J. Lambert, and J. Johnson. "A Comparison of the Outcome Questionnaire-45 and Outcome Questionnaire-30 in Classification and Prediction of Treatment Outcome." Clinical Psychology and Psychotherapy, vol. 13, 2006, pp. 380-391.

Foa, E. "Effective Treatment for PTSD, Second Edition." New York, NY: Guilford Press, 2009.

Forman-Hoffman, V., S. Knauer, J. McKeeman, A. Zolotor, R. Blanco, S. Lloyd, E. Tant, and M. Viswanathan. "Child and Adolescent Exposure to Trauma: Comparative Effectiveness of Interventions Addressing Trauma Other Than Maltreatment or Family Violence." Comparative Effectiveness Review No. 107. Report prepared by RTI International-University of North Carolina at Chapel Hill for the Agency for Healthcare Research and Quality (AHRQ). Rockville, MD: AHRQ. February 2013. Available at http://www.effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed November 8, 2013.

Garland, A.F., L. Bickman, and B.F. Chorpita. "Change What? Identifying Quality Improvement Targets by Investigating Usual Mental Health Care." Administration and Policy in Mental Health, vol. 37, no. 1-2, 2010a, pp. 15-26.

Garland, A.F., L. Brookman-Frazee, M.S. Hurlburt, E.C. Accurso, R.J. Zoffness, R. Haine-Schlagel, and W. Ganger. "Mental Health Care for Children with Disruptive Behavior Problems: A View Inside Therapists' Offices." Psychiatric Services, vol. 61, no. 8, 2010b, pp. 788-795.

Garland, A.F., M.S. Hurlburt, L. Brookman-Frazee, R.M. Taylor, and E.C. Accurso. "Methodological Challenges of Characterizing Usual Care Psychotherapeutic Practice." Administration and Policy in Mental Health, vol. 37, no. 3, 2010c, pp. 208-220.

Garland, A.F., M. Kruse, and G.A. Aarons. "Clinicians and Outcome Measurement: What's the Use?" Journal of Behavioral Health Services and Research, vol. 30, no. 4, 2003, pp. 393-405.

Glasgow, R.E., R.M. Kaplan, J.K. Ockene, E.B. Fisher, and K.M. Emmons. "Patient-Reported Measures of Psychosocial Issues and Health Behavior Should Be Added to Electronic Health Records." Health Affairs, vol. 31, no. 3, 2012, pp. 497-504.

Hannan, C., M.J. Lambert, C. Harmon, S.L. Nielsen, D.W. Smart, K. Shimokawa, and S.W. Sutton. "A Lab Test and Algorithms for Identifying Clients at Risk for Treatment Failure." Journal of Clinical Psychology, vol. 61, no. 2, 2005, pp. 155-163.

Hansen, N.B., M.J. Lambert, and E.V. Forman. "The Psychotherapy Dose-Response Effect and Its Implications for Treatment Delivery Services." Clinical Psychology: Science and Practice, vol. 9, 2002, pp. 329-343.

Harding, K.J., A.J. Rush, M. Arbuckle, M.H. Trivedi, and H.A. Pincus. "Measurement-Based Care in Psychiatric Practice: A Policy Framework for Implementation."Journal of Clinical Psychiatry, vol. 72, no. 8, 2011, pp. 1136-1143.

Hepner, K.A., F. Azocar, G.L. Greenwood, J. Miranda, and M.A. Burnam. "Development of a Clinician Report Measure to Assess Psychotherapy for Depression in Usual Care Settings." Administration and Policy in Mental Health and Mental Health Services Research, vol. 37, no. 3, 2010, pp. 221-229.

Hoagwood, K.E., P.S. Jensen, M.C. Acri, S.S. Olin., R.E. Lewamdowski, and R.K. Herman. "Outcome Domains in Child Mental Health Research Since 1996: Have They Changed and Why Does It Matter?" Journal of the American Academy of Child and Adolescent Psychiatry, vol. 51, no. 12, 2012, 1241-1260.

Hofmann, S.G., A. Asnaani, I.J. Vonk, A.T. Sawyer, and A. Fang. "The Efficacy of Cognitive Behavioral Therapy: A Review of Meta-Analyses." Cognitive Therapy and Research, vol. 36, no. 5, 2012, pp. 427-440.

Horvitz-Lennon, M., S.L. Normand, R.G. Frank, and H.H. Goldman. "'Usual Care' for Major Depression in the 1990s: Characteristics and Expert-Estimated Outcomes." American Journal of Psychiatry, vol. 160, no. 4, 2003, pp. 720-726.

Hurlburt, M.S., A.F. Garland, K. Nguyen, and L. Brookman-Frazee. "Child and Family Therapy Process: Concordance of Therapist and Observational Perspectives."Administration and Policy in Mental Health, vol. 37, no. 3, 2010, pp. 230-244.

Institute of Medicine. "Treatment for Posttraumatic Stress Disorder in Military and Veteran Populations: Initial Assessment." Washington, DC: National Academies Press, 2012.

Jakobsen, J.C., J.L. Hansen, E. Simonsen, and C. Gluud. "The Effect of Adding Psychodynamic Therapy to Antidepressants in Patients with Major Depressive Disorder. A Systematic Review of Randomized Clinical Trials with Meta-Analyses and Trial Sequential Analyses." Journal of Affective Disorders, vol. 137, no. 1, 2012, pp. 4-14.

Jonas, D.E., K. Cusack, C.A. Forneris, T.M. Wilkins, J. Sonis, J.C. Middleton, C. Feltner, D. Meredith, J. Cavanaugh, K.A. Brownley, K.R. Olmsted, A. Greenblatt, A. Weil, and B.N. Gaynes. "Psychological and Pharmacological Treatments for Adults with Posttraumatic Stress Disorder (PTSD)." Comparative Effectiveness Review No. 92. Report prepared by RTI International-University of North Carolina Evidence-based Practice Center. AHRQ Publication No. 13-EHC011-EF. Rockville, MD: Agency for Healthcare Research and Quality; April 2013. Available at http://www.effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed November 8, 2013.

Kerr, E.A., M.A. Lucatorto, R. Holleman, N.M. Hogan, M.L. Klamerus, T.P. Hofer, and VA Diabetes Quality Enhancement Research Initiative (QUERI) Workgroup on Clinical Action Measures. "Monitoring Performance for Blood Pressure Management among Diabetic Patients: Too Much of a Good Thing?" Archives of Internal Medicine, vol. 25, no. 12, 2012, pp. 938-945. Available at http://www.ncbi.nlm.nih.gov/pubmed/22641246.

Kilbourne, A.M., D. Keyser, and H.A. Pincus. "Challenges and Opportunities in Measuring the Quality of Mental Health Care." Canadian Journal of Psychiatry, Revue Canadienne De Psychiatrie, vol. 55, no. 9, 2010, pp. 549-557.

Kiresuk, T.J., A. Smith, and J.E. Cardillo (eds.). "Goal Attainment Scaling: Applications, Theory, and Measurement." Hillsdale, NJ: Lawrence Erlbaum Associates, Inc., 1994.

Knaup, C., M. Koesters, D. Schoefer, T. Becker, and B. Puschner. "Effect of Feedback of Treatment Outcome in Specialist Mental Healthcare: Meta-Analysis."British Journal of Psychiatry: Journal of Mental Science, vol. 195, no. 1, 2009, pp. 15-22.

Kraus, D.R. "The Treatment Outcome Package (TOP)." Integrating Science and Practice, vol. 2, no. 2, 2012, pp. 43-45.

Lambert, M.J., C. Harmon, K. Slade, J.L. Whipple, and E.J. Hawkins. "Providing Feedback to Psychotherapists on Their Patients' Progress: Clinical Results and Practice Suggestions." Journal of Clinical Psychology, vol. 61, no. 2, 2005, pp. 165-174.

Lambert, M.J., and D.E. Barley. (2002). "Research Summary on the Therapeutic Relationship and Psychotherapy Outcome." In Psychotherapy Relationships that Work: Therapist Contributions and Responsiveness to Patients, edited by J.C. Norcross. Oxford: Oxford University Press.

Lambert, M.J., J.L. Whipple, M.J. Bishop, D.A. Vermeersch, G.V. Gray, and A.E. Finch. "Comparison of Empirically Derived and Rationally Derived Methods for Identifying Clients at Risk for Treatment Failure." Clinical Psychology and Psychotherapy, vol. 9, 2002, pp. 149-164.

Laska, K.M., A.S. Gurman, and B.E. Wampold. "Expanding the Lens of Evidence-Based Practice in Psychotherapy: A Common Factors Perspective."Psychotherapy, online ahead of print. doi: 10.1037/a0034332.

Leichsenring, F. "Are Psychodynamic and Psychoanalytic Therapies Effective?: A Review of Empirical Data." International Journal of Psychoanalysis, vol. 86, 2005, pp. 841-868.

Leichsenring, F., and E. Leibing. "Psychodynamic Psychotherapy: A Systematic Review of Techniques, Indications, and Empirical Evidence." Psychology and Psychotherapy, vol. 80, 2007, pp. 217-228.

Liptzin. B. "Quality Improvement, Pay for Performance, and Outcomes Measurement: What Makes Sense?" Psychiatric Services, vol. 60, 2009, pp. 108-111.

Long, M.E., A.L. Grubaugh, J.D. Elhai, K.J. Cusack, R. Knapp, and B.C. Frueh. "Therapist Fidelity with an Exposure-Based Treatment of PTSD in Adults with Schizophrenia or Schizoaffective Disorder." Journal of Clinical Psychology, vol. 66, no. 4, 2010, pp. 383-393.

Lu, W., P.T. Yanos, J.D. Gottlieb, S.M. Duva, S.M. Silverstein, H. Xie, S.D. Rosenberg, and K.T. Mueser. "Use of Fidelity Assessments to Train Clinicians in the CBT for PTSD Program for Clients with Serious Mental Illness." Psychiatric Services, vol. 63, no. 8, 2012, pp. 785-792.

Maalouf, F.T., and D.A. Brent. "Child and Adolescent Depression Intervention Overview: What Works, for Whom and How Well?" Child and Adolescent Psychiatric Clinics of North America, vol. 21, no. 2, 2012, pp. 299-312.

Markowitz, J.C., and M.M. Weissman. "Interpersonal Psychotherapy: Principles and Applications." World Psychiatry : Official Journal of the World Psychiatric Association, vol. 3, no. 3, 2004, pp. 136-139.

Martino, S., S. Ball, C. Nich, T.L. Frankforter, and K.M. Carroll. "Correspondence of Motivational Enhancement Treatment Integrity Ratings Among Therapists, Supervisors, and Observers." Psychotherapy Research, vol. 19, no. 2, 2009, pp. 181-193.

Miranda, J., K.A. Hepner, F. Azocar, G. Greenwood, V. Ngo, and M.A. Burnam. "Development of a Patient-Report Measure of Psychotherapy for Depression."Administration and Policy in Mental Health and Mental Health Services Research, vol. 37, no. 3, 2010, pp. 245-253.

McAleavey, A.A., S.S. Nordberg, D.R. Kraus, and L.G. Castonguay. "Errors in Treatment Outcome Monitoring: Implications of Multidimensional and General Measurements for Real-World Psychotherapy." Canadian Psychology, vol. 53, 2012, pp. 105-114.

McHugh, R.K., H.W. Murray, and D.H. Barlow. "Balancing Fidelity and Adaptation in the Dissemination of Empirically-Supported Treatments: The Promise of Transdiagnostic Interventions." Behaviour Research and Therapy, vol. 47, no. 11, 2009, pp. 946-953.

Mohr, D.C., M.N. Burns, S.M. Schueller, G. Clarke, and M. Klinkman. "Behavioral Intervention Technologies: Evidence Review and Recommendations for Future Research in Mental Health." General Hospital Psychiatry, vol. 35, no. 4, 2013, pp. 332-338.

Moss, S., F. Margison, and K. Godbert. "The Maintenance of Psychotherapy Skill Acquisition: A Two-Year Follow-Up." British Journal of Medical Psychology, vol. 64, 1991, pp. 233-236.

National Committee for Quality Assurance (NCQA). "Integrated Care for People with Medicare and Medicaid: A Roadmap for Quality." March 2013. Available athttp://thescanfoundation.org/sites/thescanfoundation.org/files/ncqa-integrated_care_for_people_medicare_medicaid-3-6-13.pdf. Accessed July 12, 2013.

National Institute for Health and Care Excellence. "Post-Traumatic Stress Disorder. The Management of PTSD in Adults and Children in Primary and Secondary Care." CG 26. London: National Institute for Health and Care Excellence, 2005.

National Institute for Health and Care Excellence. "Depression in Adults. The Treatment and Management of Depression in Adults." CG 90. London: National Institute for Health and Care Excellence, 2009.

National Institute for Health and Care Excellence. "Generalised Anxiety Disorder and Panic Disorder (with or Without Agoraphobia) in Adults. Management in Primary, Secondary and Community Care." CG 113. London: National Institute for Health and Care Excellence, 2011.

National Quality Forum. "NQF-Endorsed Measures (QPS)." Available at http://www.Qualityforum.org/QPS/QPSTool.aspx. Accessed April 28, 2014.

National Quality Forum. "Patient Reported Outcomes (PROs) in Performance Measurement." January, 2013. Available athttp://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx. Accessed November 18, 2013.

National Quality Forum. "Measuring Healthcare Quality for the Dual Eligible Beneficiary Population." June 2012. Available athttp://www.qualityforum.org/Publications/2012/06/Measuring_Healthcare_Quality_for_the_Dual_Eligible_Beneficiary_Population.aspx. Accessed February 13, 2014.

Okiishi, J.C., M.J. Lambert, D. Eggett, L. Nielsen, D.D. Dayton, and D.A. Vermeersch. "An Analysis of Therapist Treatment Effects: Toward Providing Feedback to Individual Therapists on Their Clients' Psychotherapy Outcome." Journal of Clinical Psychology, vol. 62, no. 9, 2006, pp. 1157-1172.

Olfson, M., and S.C. Marcus. "National Trends in Outpatient Psychotherapy." American Journal of Psychiatry, vol. 167, no. 12, 2010, pp. 1456-1463.

Otte, C. "Cognitive Behavioral Therapy in Anxiety Disorders: Current State of the Evidence." Dialogues in Clinical Neuroscience, vol. 13, no. 4, 2011, p. 413.

Pfeiffer, N., J. Glass, K. Austin, M. Valenstein, J.F. McCarthy, and K. Zivin. "Impact of Distance and Facility of Initial Diagnosis on Depression Treatment." Health Services Research, vol. 46, no. 3, 2011, pp.768-786.

Ramsay, J.R. "Current Status of Cognitive-Behavioral Therapy as a Psychosocial Treatment for Adult Attention-Deficit/Hyperactivity Disorder." Current Psychiatry Reports, vol. 9, no. 5, 2007, pp. 427-433.

Reese, R.J., M.D. Toland, N.C. Slone, and L.A. Norsworthy. "Effect of Client Feedback on Couple Psychotherapy Outcomes." Psychotherapy: Theory, Research, Practice, Training, vol. 47, no. 4, 2010, pp. 616-630. Available at http://www.ncbi.nlm.nih.gov/pubmed/21198247.

Robinson, R. "Attachment O: Cognitive Behavioral Therapy Competency Policy." 2012. Available at http://www.dshs.state.tx.us/mhsa/trr/documents/. Accessed July 10, 2013.

Rost, K., J. Fortney, and J. Coyne. "The Relationship of Depression Treatment Quality Indicators to Employee Absenteeism." Mental Health Services Research, vol. 7, no. 3, 2005, pp. 161-169.

SAMHSA-HRSA Center for Integrated Health Solutions. "Behavioral Health Homes for People with Mental Health and Substance Use Conditions: The Core Clinical Features." May 2012. Available at http://www.integration.samhsa.gov/resource/behavioral-health-homes-for-people-with-mental-health-substance-use-conditions-the-core-clinical-features. Accessed June 15, 2013.

Schoenwald, S.K., and A.F. Garland. "A Review of Treatment Adherence Measurement Methods." Psychological Assessment, vol. 25, no. 1, 2013, pp. 146-156.

Schoenwald, S.K., A.F. Garland, J.E. Chapman, S.L. Frazier, A.J. Sheidow, and M.A. Southam-Gerow. "Toward the Effective and Efficient Measurement of Implementation Fidelity." Administration and Policy in Mental Health, vol. 38, no. 1, 2011a, pp. 32-43.

Schoenwald, S.K., A.F. Garland, M.A. Southam-Gerow, B.F. Chorpita, and J.E. Chapman. "Adherence Measurement in Treatments for Disruptive Behavior Disorders: Pursuing Clear Vision Through Varied Lenses." Clinical Psychology, vol. 18, no. 4, 2011b, pp. 331-341.

Schoenwald, S.K. "It's a Bird, It's a Plane, It's...Fidelity Measurement in the Real World." Clinical Psychology: A Publication of the Division of Clinical Psychology of the American Psychological Association, vol. 18, no. 2, 2011c, pp. 142-147.

Schoenwald, S.K., J.E. Chapman, A.J. Sheidow, and R.E. Carter. "Long-Term Youth Criminal Outcomes in MST Transport: The Impact of Therapist Adherence and Organizational Climate and Structure." Journal of Clinical Child and Adolescent Psychology, vol. 38, no. 1, 2009, pp. 91-105.

Shedler, J. "The Efficacy of Psychodynamic Psychotherapy." American Psychologist, vol. 65, no. 2, 2010, pp. 98-109.

Shimokawa, K, M.J. Lambert, and D.W. Smart. "Enhancing Treatment Outcome of Patients at Risk for Treatment Failure: Meta-Analytic and Mega-Analytic Review of a Psychotherapy Quality Assurance System." Journal of Consulting and Clinical Psychology, vol. 78, no. 3, 2010, pp. 298-311.

Shiner, B., L.W. D'Avolio, T.M. Nguyen, M.H. Zayed, Y. Young-Xu, R.A. Desai, P.P. Schnurr, L.D. Fiore, and B.V. Watts. "Measuring Use of Evidence-Based Psychotherapy for Posttraumatic Stress Disorder." Administration and Policy in Mental Health, vol. 40, no. 4, 2013, pp. 311-318.

Simon, G.E., V. Ding, R. Hubbard, P. Fishman, E. Ludman, L. Morales, B. Operskalski, and J. Savarino. "Early Dropout from Psychotherapy for Depression with Group- and Network-Model Therapists." Administration and Policy in Mental Health, vol. 39, no. 6, 2012a, pp. 440-447.

Simon, G.E., Z.E. Imel, E.J. Ludman, and B.J. Steinfeld. "Is Dropout After a First Psychotherapy Visit Always a Bad Outcome?"Psychiatric Service, vol. 63, no. 7, 2012b, pp. 705-707.

Stiles, W.B., C. Leach, M. Barkham, M. Lucock, S. Iveson, D.A. Shapiro, M. Iveson, and G.E. Hardy. "Early Sudden Gains in Psychotherapy Under Routine Clinic Conditions: Practice-Based Evidence." Journal of Consulting and Clinical Psychology, vol. 71, no. 1, 2003, pp. 14-21.

Stirman, S.W., A. Gutierrez-Colina, K. Toder, G. Esposito, F. Barg, F. Castro, A.T. Beck, and P. Crits-Christoph. "Clinicians' Perspectives on Cognitive Therapy in Community Mental Health Settings: Implications for Training and Implementation." Administration and Policy in Mental Health, vol. 40, no. 4, 2013, pp. 274-285.

Sue, S., N. Zane, H.C. Nagayama, and L.K. Berger. "The Case for Cultural Competency in Psychotherapeutic Interventions." Annual Review of Psychology, vol. 60, 2009, pp. 525-548.

Teague, G.B., K.T. Mueser, and C.A. Rapp. "Advances in Fidelity Measurement for Mental Health Services Research: Four Measures." Psychiatric Services, vol. 63, no. 8, 2012, pp. 765-771.

Torda, P. "Measures Using Patient-Reported Outcomes." Presented at the PCORI Patient Reported Outcome Workshop, Atlanta, GA. November 19, 2013. Available at http://www.pcori.org/assets/2013/11/PCORI-PRO-Workshop-Presentation-Panel-II-111913.pdf. Accessed February 16, 2014.

van Hees, M.L., T. Rotter, T. Ellermann, and S.M. Evers. "The Effectiveness of Individual Interpersonal Psychotherapy as a Treatment for Major Depressive Disorder in Adult Outpatients: A Systematic Review." BMC Psychiatry, vol. 13, no. 1, 2013, p. 22.

Warren, J.S., P.L. Nelson, S.A. Mondragon, S.A. Baldwin, and G.M. Burlingame. "Youth Psychotherapy Change Trajectories and Outcomes in Usual Care: Community Mental Health Versus Managed Care Settings." Journal of Consulting and Clinical Psychology, vol. 78, 2012, pp. 144-155.

Watkins, K.E., M. Horvitz-Lennon, L.B. Caldarone, L.R. Shugarman, B. Smith, T.E. Mannle, D.R. Kivlahan, and H.A. Pincus. "Developing Medical Record-Based Performance Indicators to Measure the Quality of Mental Healthcare." Journal of Healthcare Quality, vol. 33, no.1, 2011, pp. 49-66.

Watkins, K.E., D.J. Keyser, B. Smith, T.E. Mannle, D.R. Kivlahan, S.M. Paddock, T. Mattox, M. Horvitz-Lennon, and H.A. Pincus. "Transforming Mental Healthcare in the Veterans Health Administration: A Model for Measuring Performance to Improve Access, Quality, and Outcomes." Journal for Healthcare Quality, vol. 32, no. 6, 2010, pp. 33-42.

Wei, W., U. Sambamoorthi, M. Olfson, J.T. Walkup, and S. Crystal. "Use of Psychotherapy for Depression in Older Adults." American Journal of Psychiatry, vol. 162, no. 4, 2005, pp. 711-717.

Weissman, M.M., H. Verdeli, M.J. Gameroff, S.E. Bledsoe, K. Betts, L. Mufson, H. Fitterling, and P. Wickramarantne. "National Survey of Psychotherapy Training in Psychiatry, Psychology, and Social Work." Archives of General Psychiatry, vol. 63, no. 8, 2006, pp. 925-934.

Weisz, J.R., A. Jensen-Doss, and K.M. Hawley. "Evidence-Based Youth Psychotherapies Versus Usual Clinical Care: A Meta-Analysis of Direct Comparisons."American Psychologist, vol. 61, no. 7, 2006, pp. 671-689.

Weisz, J.R., A.M. Ugueto, D.M. Cheron, and J. Herren. "Evidence-Based Youth Psychotherapy in the Mental Health Ecosystem." Journal of Clinical Child and Adolescent Psychology, vol. 42, no. 2, 2013, pp. 274-286.

Whipple, J.L., and M.J. Lambert. "Outcome Measures for Practice." Annual Review of Clinical Psychology, vol. 7, 2011, pp. 87-111.

Wissow, L.S., J.D. Brown, K. Fothergill, A. Gadomski, K. Hacker, P. Salmon, and R. Zelkowitz. "Framing and Discussing Universal Mental Health Screening in Pediatric Primary Care: A Systematic Review." Journal of the American Academy of Child and Adolescent Psychiatry, in press.

Youn, S.J., D.R. Kraus, and L.G. Castonguay. "The Treatment Outcome Package: Facilitating Practice and Clinically Relevant Research." Psychotherapy, vol. 49, 2012, pp. 115-122.

Appendix A. Clinical Guideline Definitions of Evidence Ratings

This report was prepared under contract #HHSP2332010016WI between the U.S. Department of Health and Human Services (HHS), Office of Disability, Aging and Long-Term Care Policy (DALTCP) and Mathematica Policy Research. For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/office_specific/daltcp.cfm or contact the ASPE Project Officer, D.E.B. Potter, at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. His e-mail address is: D.E.B.Potter@hhs.gov.

To obtain a printed copy of these reports, send the full report title and your mailing information to:

U.S. Department of Health and Human Services
Office of Disability, Aging and Long-Term Care Policy
Room 424E, H.H. Humphrey Building
200 Independence Avenue, S.W.
Washington, D.C. 20201
FAX:  202-401-7733
Email:  webmaster.DALTCP@hhs.gov