Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation. III. Evidence-based Psychotherapy in Practice and Options for Quality Measures


Efficacy trials of psychotherapy typically include a certain number of visits or an expected progression through phases of treatment. As psychotherapies move from efficacy trials to practice, the content and duration of treatment is often adapted to the provider's clinical orientation or the context of the treatment setting. In addition, treatments may be tailored to the consumer's illness severity, past experience with treatment or other characteristics. Adhering to the treatment protocols tested in trials may be challenging for a variety of practical reasons.

There is also strong evidence that mental health providers tend to adopt "eclectic" treatment approaches that draw from different theoretical orientations and professional training (Garland et al. 2010a). Rarely does a provider only offer one type of psychotherapy (CBT, for example) and the provider may blend techniques from different types of psychotherapy to meet the consumer's needs. There is also evidence that providers tend to use some therapeutic skills more than others. For example, one study that videotaped psychotherapy sessions of children found that providers delivered some key elements of treatment frequently (for example, positive reinforcement and psychoeducation) but other components were less common (for example, role-playing or assigning homework) (Garland et al. 2010b). Another study found that when providers were asked about their perceptions of cognitive therapy before they learned how to use it, many of them reported that they planned to selectively implement elements of the therapy rather than attempt to deliver the full protocol (Stirman et al. 2013). Consequently, the care delivered in typical community-based treatment settings might not resemble the psychotherapy tested in trials (Brookman-Frazee et al. 2008, 2010; Baumann et al. 2006). Such adaptations are understandable and may facilitate the wider dissemination of the treatment (McHugh et al. 2009; Chorpita and Regan 2009), but researchers do not have a good understanding of how specific adaptations in the duration or content of psychotherapy affect outcomes. Although there is good evidence to suggest that the duration of treatment in the community is rarely consistent with trials (Simon et al. 2012a), there is very little information on the content of psychotherapy delivered outside of research settings (Garland et al. 2010c).

When care in the community has been studied, researchers have noted that evidence-based psychotherapies implemented in typical community-based treatment settings produce more modest outcomes than efficacy trials (Weisz et al. 2013; Weisz et al. 2006). This may be because care in the community diverges from the protocols tested in trials; it may also relate to the broader population of consumers included in community-based treatment settings (where efficacy trials tend to have many exclusion criteria). This observation has several implications for quality measurement. Efforts to advance the adoption of evidence-based psychotherapies are likely to require substantial changes in practice for many providers, payers, and consumers. Making these changes is likely to require intensive support for provider training and consumer engagement as well as new payment models that promote evidence-based practice. Confidence in the anticipated outcomes of psychotherapy is critical to both the successful implementation of quality measures and the spread of evidence-based psychotherapy.

To guide our consideration of the possible approaches to measuring the quality of psychotherapy for the purposes of accountability and quality improvement, we used the Donabedian (1980, 1988) quality framework to describe measures of structures, processes, and outcomes. In Table III-1, we summarize the measure concepts, potential data sources, and strengths and limitations of each type of measure. Structure measures evaluate the organization--including such resources as the staffing, equipment, and protocols. For psychotherapy, structure measures would focus on whether providers have the capacity to deliver evidence-based care and to monitor outcomes--for example, whether providers have appropriate training, use treatment protocols and tools consistent with psychotherapies tested in trials, and monitor symptoms and functioning using standardized tools. Process measures describe how structures are put into place and are quantified for consumers (for example, process measures would allow us to determine the extent to which individuals received psychotherapy that is consistent with the content of treatment that tested in trials). Because treatments are adapted to fit into service settings, process measures tell us something about the extent to which these adaptations result in the delivery of treatment that still contains the core components of the evidence-based model. Finally, outcome measures assess the end results of processes; such measures would examine whether the delivery of psychotherapy reduces the severity of a consumer's symptoms and improves his or her functioning.

TABLE III-1. Strategies to Measure the Quality of Psychotherapy
Type of
  Strengths     Challenges  
Structure   Capability for delivering evidence-based psychotherapy and measuring outcomes Documentation submitted by provider

On-site audits including consumer or staff interviews

Develop capacity among providers to implement treatments

Potential for use in credentialing and payment (e.g., provider could be "qualified" to bill for specific services or to receive incentive payment for having/demonstrating capability)

Good tool for bringing the field along

Could build on existing fidelity tools used for training and program improvement

Interim approach; need to get to process and outcome measures

Documentation/auditing is burdensome

Training to meet requirements is expensive 

Potential for gaming/ providing socially desirable responses

Process Access/frequency of visits Claims Low-cost/low-burden approach to measurement

Building block measure

Provide little information on content of visits

Coding of services varies across health plans and state Medicaid programs

Relationship of number of visits to evidence base is unknown

May promote utilization without strong evidence, which goes against prevailing concerns about costs of care

Process Documentation of evidence-based treatments in medical record/EHRs Medical records or EHR Opportunity to build template for documenting key elements into medical records/EHRs 

Data could be available for supervision and quality improvement

Current limited use of EHR in mental health care 

New documentation burden for providers 

Potential to become checkbox measures 

Difficult and costly to access paper records 

Processes are not well-defined

Process Consumer-reported and provider-reported content of psychotherapy Surveys Could build on existing assessments in areas like CBT 

Provider surveys could complement structure measures 

Consumer surveys focus on what consumers remember/get from services

Expensive to field surveys 

Challenging to sample consumers and providers 

Some consumers may not be reliable reporters; proxies may be necessary 

Validity of consumer responses is uncertain 

Potential for gaming/ providing socially desirable responses among providers 

Need for strategies to determine the period of assessment and the measure score

Outcome Outcomes reported by consumers, family members, and others Standardized tools that assess improvements in symptom severity and functioning; could be built into or stored in EHRs or other electronic systems (e.g., web-based tracking systems) Focus on outcomes that are important to consumers and families 

Provides information that could be used for clinical decision making and quality improvement 

Opportunity to build goal assessment and achievement measures 

More flexible approach than specifying processes

Requires new workflows and data collection processes; providers would need training and support to administer assessments and use feedback from assessments 

Need to identify a common set of standardized tools that can apply to broad populations 

Need to develop measures that allow for comparisons across providers, health plans, states, and other accountable entities 

Need for risk adjustment strategies 

Some consumers may have difficulty reporting; proxies may be necessary


View full report


"QualPsy.pdf" (pdf, 445.96Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®