Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation. VI. Outcome Measures


Outcome measures would assess whether individuals receiving psychotherapy experience improvements in their symptoms and functioning across a broad range of life domains, including employment, school functioning, relationships, and engagement in the community (Hoagwood et al. 2012). The use of such measures would complement efforts already underway to encourage the measurement of patient-reported outcomes in other areas of health care. For example, CMS and the Office of the National Coordinator for Health Information Technology are supporting efforts to develop patient-reported outcome measures for use in the future stages of the CMS EHR incentive program. These measures address medical and behavioral health conditions. In addition, HHS recently supported a NQF project that sought to identify the factors that should be considered for selecting patient-reported outcome measures for performance improvement and accountability (NQF 2013). Finally, the ACA established the Patient-Centered Outcomes Research Institute to conduct comparative effectiveness research focused on outcomes important to patients. Many of these projects employ or are testing measurement strategies that offer feedback to providers and consumers in an effort to support their ability to make treatment decisions.

As in other areas of health care, measuring psychotherapy outcomes could involve the use of repeated assessments to track improvements over time and progress toward reaching the consumers' goals. These assessments could be completed directly by the consumer. With consumer consent, family members or other individuals important to the consumer, such as case managers or teachers (particularly for children or adolescents receiving therapy), could also complete assessments since these reporters may offer different perspectives (Brown et al. 2008, 2007). Providers could use information from these repeated assessments to adjust treatment in response to an individual's progress, and the individual receiving treatment can use the information for self-monitoring and to make treatment decisions (for example, inquiring about more intensive treatment options or changing providers). Such data could be stored in medical records/EHRs or in other electronic systems, such as web-based systems. In the aggregate, health systems can use the data from these repeated assessments to monitor how consumers respond to treatment and to identify opportunities for quality improvement. Such data can facilitate comparisons of outcomes across these entities. This approach of using repeated assessments to measure outcomes, tailor treatment, and improve the quality of care--referred to as measurement-based care or routine outcomes monitoring in the literature (Boswell et al. 2013; Harding et al. 2011)--is common for physical health conditions, such as diabetes and hypertension, for which measurements are taken regularly and treatment is adjusted accordingly.

There are several examples of measurement-based care or routine outcome monitoring systems in mental health care (Drapeau 2012). Here we provide a brief description of some of these systems to illustrate how they work in practice.

Perhaps one of the most well-known outcomes monitoring efforts in mental health care is the Depression Improvement Across Minnesota Offering A New Direction (DIAMOND) project, in which participating health plans pay certified practices a flat monthly rate for providing a bundled set of services for depression or dsythymia. As part of the initiative, practices administer the PHQ-9 during the consumer's first visit and again at six months and 12 months after the initial visit (AHRQ 2013b). Practices receive monthly performance reports that include how many consumers completed the PHQ-9, symptom remission rates, and how many consumers are making progress toward feeling better (defined as at least 50 percent reduction in the baseline PHQ-9 score). These measures were incorporated into the work of Minnesota Community Measurement, which maintains a website that publicly reports these measures for clinics participating in DIAMOND and other primary care and behavioral health clinics across the state ( The website facilitates comparisons of clinics over time and reports state averages. The Minnesota measures are endorsed by NQF and included in the CMS EHR Incentive programs. Other delivery systems (Kaiser Permanente and Group Health, for example) and community initiatives (MaineHealth, for example) are adopting these measures and similar measurement and reporting systems.

Other health plan initiatives have also used repeated outcome assessments that give feedback to providers and/or consumers. For example, Optum Behavioral Health, a large national managed behavioral health organization with more than 100,000 providers in its network, uses the Algorithms for Effective Reporting and Treatment (ALERT) system. This system combines data from a consumer-reported Wellness Assessment with claims data to track consumer improvement and identify individuals who are at risk for poor outcomes. The ALERT system identifies consumers with "high distress" or who are at risk of substance abuse who demonstrate poor progress early in treatment. The one-page Wellness Assessment contains items derived from validated tools that assess symptom severity, functional impairment, self-efficacy, substance abuse risk, and the presence of co-morbid medical conditions. The provider administers the assessment when treatment begins and then again during later visits. With permission from the consumer, Optum mails a follow-up assessment four months after treatment begins.

Several web-based systems for tracking symptoms and functioning have been used for quality monitoring and improvement in mental health care settings. One example is the Treatment Outcome Package (TOP), which tracks mental health symptoms and functioning across 12 clinical domains (Kraus 2012). Providers use this system to email a link to the consumer to complete an online questionnaire (which requires 3-5 minutes). The system scores the questionnaire and generates a short report for the provider. Over time, these reports graphically display changes in scores within each domain and benchmark those scores to the general non-clinical population. The report alerts the provider if the consumer is not making progress as expected and includes a list of suggested treatment practices aimed at improving outcomes. TOP also generates a section of the report designed to give to the consumer as feedback. Providers also receive monthly aggregate reports that benchmark their risk adjusted performance against similar professionals. Health care systems have used this system widely. For example, Blue Cross and Blue Shield of Massachusetts incentivized the use of TOP by requiring that providers achieve certain response rates on the tool in order to receive their annual provider fee increase. This was not without controversy and pushback from providers, but ultimately, the TOP was administered over 40,000 times in the first six months of the program (Youn et al. 2012; Liptzin 2009; Blais et al. 2009).

Another example of an approach to outcomes monitoring is the Partners for Change Outcome Management System (PCOMS) International Center for Clinical Excellence, which was recently listed in the SAMHSA National Registry of Evidence-based Programs and Practices (Reese et al. 2010; Anker et al. 2009; Campbell and Hemsley 2009). PCOMS consists of two brief scales: (1) the Outcome Rating Scale (ORS), which assesses mental health functioning and distress and the consumer's perceived benefit of treatment; and (2) the Session Rating Scale (SRS), which assesses the consumer's perception of the therapeutic alliance. The provider administers the ORS at the beginning of the therapy session and the SRS toward the end of the session. The provider and consumer discuss the consumer's ratings for both measures on a session-by-session basis to encourage the consumer's engagement in treatment, improve therapeutic alliance, and keep the sessions focused on the concerns identified by the consumer.

A final example of routine outcomes monitoring is the Improving Access to Psychological Therapy program, which currently operates throughout much of England (Department of Health 2012). This treatment model contains several components, including the use of assessments that identify the individual's concerns and treatment goals at initial contact, and tracks symptom reduction and progress toward treatment goals. Participating providers must ensure that at least 90 percent of consumers who are seen at least twice receive pre-treatment and post-treatment assessments and have a score on the main outcome measures. In addition, these providers receive weekly feedback and clinical supervision to discuss adjusting treatment based on information from the assessments. Information from these assessments, as well as other clinical information, is stored in an electronic database that therapists and care managers can access to monitor consumer progress and managers can use to monitor care and identify opportunities for quality improvement.

The measurement strategies described above, as well as others, have demonstrated promising results. Several randomized trials have found that assessing symptoms and functioning at regular intervals and giving the results as feedback to providers helps to identify individuals at risk for poor outcomes, prevents the worsening of symptoms, and decreases the time to positive outcomes (Bickman et al. 2011; Whipple and Lambert 2011; Shimokawa et al. 2010; Lambert et al. 2005). One meta-analysis of trials that examined feedback given to mental health providers during the course of treatment found a modest positive effect on short-term consumer outcomes but no effect on treatment duration, costs, or longer-term consumer outcomes (although very few studies included information on treatment costs or duration) (Knaup et al. 2009). The same meta-analysis found that feedback to providers had a larger positive effect on short-term outcomes if: (1) the feedback included information on mental health progress over time (versus providing information about current status only); (2) both the consumer and provider received feedback (versus only one of them); and (3) feedback was given more than once. These findings are consistent with the experience of one managed behavioral health organization, which found that six-month outcomes were better among consumers whose therapist reported using the information provided in the progress reports compared with consumers whose provider received the reports but did not report using them (Azocar et al. 2007). These findings underscore the importance of having user-friendly mechanisms that enable providers and consumers to use the feedback from outcome measures.

View full report


"QualPsy.pdf" (pdf, 445.96Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®