Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation. B. Challenges


Health systems must overcome several obstacles to widely implement psychotherapy outcome measures, including: (1) selecting outcome measures that are meaningful for consumers, providers, and other stakeholders; (2) deciding on the appropriate level of reporting and strategies for making fair comparisons across providers, plans, or systems; (3) overcoming providers' lack of familiarity with outcome measures; and (4) incorporating the administration and reporting of measures into clinical workflows so that it becomes a routine part of therapy (Boswell et al. 2013; Harding et al. 2011). Here we briefly discuss these challenges.

Selecting appropriate outcome assessments. There are many choices of outcome assessments. Some assessments focus on symptom severity among specific diagnostic groups; others focus more broadly on functioning across various life domains. Some of these assessments are proprietary. The selection of the best assessment may depend on the target population, treatment setting, and end use of the information. Although it is important to have some flexibility in which instruments are used to assess outcomes, the use of very different assessments across providers, health plans, or states may impede comparisons. In addition, because providers typically belong to several health plans and receive reimbursement through various state and federal funding streams--each with their own reporting requirements--aligning outcome reporting efforts would decrease burden.

In addition to measuring symptoms and functioning, experts and measurement stakeholders, including the Measures Application Partnership Dual Eligible Beneficiaries Workgroup, have recommended that performance measures should assess goal setting and goal achievement (NQF 2012). For example, the framework guiding measure-development for the CMS EHR incentive program calls for steps in building performance measures based on patient-reported outcomes. Experts recommended gradually introducing performance measures that include goal setting, goal attainment, and improvement in outcomes over time (Torda 2013). This goal setting approach is drawing on literature on goal attainment scaling developed in the mental health field and implemented in a variety of settings and populations (Kiresuk et al. 1994).

Ideally, outcome assessments would be inexpensive to collect, impose the least amount of data collection burden to providers and consumers, apply to broad populations, and have some comparability in terms of reliability and validity. Stakeholders could draw from a common menu of outcome assessments that would facilitate comparisons. There are some resources on which to build; the MacArthur Foundation and Project IMPACT have undertaken efforts to develop a tool kit of measures for depression (Harding et al. 2011), and the Patient-Reported Outcomes Measurement Information System initiative, sponsored by the HHS National Institutes of Health, has assembled measures and items that assess symptoms and functioning. When these measures are used for quality improvement within an organization, the goal may be simply improvement without a specific target. However, if different measures were used across providers or health plans for the purposes of public reporting and accountability, health plans or other entities would need to use methods to make equitable comparisons. Statistical concepts, such as the Reliable Change Index, could be used.

Lack of familiarity with outcomes monitoring. Historically, academic training for psychiatrists and other behavioral health providers has not included learning how to incorporate outcome assessments into clinical practice. Some providers may be reluctant to use assessments because they fear it could damage their relationships with consumers or threaten their autonomy (Boswell et al. 2013). Providers may also not see the value of using repeated assessments based on standardized scales (Lambert et al. 2005) or may not want their outcomes compared with other therapists (Youn et al. 2012; Okiishi et al. 2006).

Providers would need training to learn how to introduce assessments to consumers, interpret the results, and use those results for clinical decision making and quality improvement. There is surprisingly little research to guide how providers should approach the steps involved in introducing and administering mental health assessment tools and discussing the results of those tools with consumers (Wissow et al. 2013).

Getting providers to use feedback can be a formidable challenge. One study found that even when outcome assessments were required, most mental health providers reported that they did not use the feedback for clinical decision making (Garland et al. 2003). It is unclear what strategies work best for providing feedback and how to structure that feedback in a manner that is helpful. Most outcome measurement and feedback strategies in psychotherapy do not give specific instruction on how providers should use feedback, allowing providers to use their clinical judgment instead (Lambert et al. 2005). Some feedback strategies have successfully used color-coded systems that correspond to consumer progress, and there is some evidence that certain clinical support tools can help providers use feedback, but further research is needed to understand how these strategies and tools can work in typical community-based treatment settings (Whipple and Lambert 2011). Health plans have employed various strategies for giving feedback to providers. For example, PacifiCare Behavioral Health sent quarterly reports to providers that contained a summary of progress for their patient population (Brown and Jones 2005). They also sent letters to providers when a consumer failed to demonstrate improvement, which encouraged the provider to keep that individual engaged in treatment and offered to pre-authorize more intensive services. They also sent letters to providers when individuals responded well to treatment; the idea was to acknowledge the good outcome and suggest that longer-term treatment might not necessarily result in a better outcome. It is unclear to what extent providers may perceive such feedback as limiting their autonomy or attempting to restrict access to care--potentially adding to their reservations for participating in outcome measurement systems.

Making outcome measurement part of routine care. Providers and health plans will need to adopt new processes for measuring consumer outcomes and for using this information to improve the quality of care. Some providers may not have the time or resources to administer, score, or interpret assessments. Reimbursement models that pay for the administration of such outcome measures in mental health, similar to routine medical tests, may encourage their use (Boswell et al. 2013). There may also be opportunities to integrate these measures into existing reporting programs, such as Meaningful Use, which would provide a financial incentive. Moreover, providers will be challenged to incorporate the use of measures into their routine practice as opposed to an additional activity. Some providers with limited time may be able to rely on non-clinical staff, such as medical assistants or care managers, to administer and score the measures. Web-based or computer-based screening tools could ease the administration of measures, but smaller practices might not find investments in these technologies feasible. Health plans or large health care delivery systems may be well-positioned to provide the infrastructure to facilitate the measurement of outcomes and give feedback to providers, as in the examples described above.

Making fair comparisons. If outcome measures are used for public reporting or accountability they may require risk adjustment for two reasons: (1) to ensure that performance is not attributable to differences in severity of illness or other factors that may be beyond the control of the provider, organization, health plan, etc.; and (2) to guard against the possibility that providers or health plans would have an incentive to avoid treating/enrolling consumers with more severe problems. Methods for risk adjustment in mental health care are limited, in part because of the incomplete data on severity of illness and other consumer characteristics in claims or medical records. Some of the outcome measurement systems described in this paper have employed risk adjustment strategies that may offer guidance, whereas others are proprietary. Some potential alternatives to risk adjustment include reporting on stratified populations or examining whether there was meaningful change in clinical care in response to lack of improvement (Kerr et al. 2012). In addition, states, health plans, and providers could use measures to monitor incremental improvements rather than absolute values (Kilbourne et al. 2010). For example, a provider or health plan would be held accountable for whether consumer outcomes are improving or meeting a benchmark from one year to the next rather than being assessed and compared at only one point in time. Another possible variation might be to provide incentives to organizations that can demonstrate clinical improvement for a minimally acceptable percentage of their consumers rather than require improvement across the entire cohort.

View full report


"QualPsy.pdf" (pdf, 445.96Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®