Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation. C. Consumer and Provider Reports of the Content of Psychotherapy


Measures of whether specific psychotherapy content was delivered could be based on data collected through consumer or provider–reported surveys. When used for accountability purposes, such measures would have to consider the optimal timing and recall periods and the ability of consumers and providers to accurately report the content of treatment. In this section, we describe the strengths and limitations of measures that require consumers or providers to report the content of treatment; in the next chapter we discuss consumer-reported outcomes.

1. Advantages Associated with Consumer-Reported Measures

Consumer-reported measures provide an opportunity to assess the delivery of care from the perspective of the person receiving and ultimately impacted by the care. Such measures could also provide a mechanism for consumers to engage providers in discussions about how treatment will be provided and the rationale behind the treatment. For example, completing a measure at some point during treatment may encourage the consumer to ask for certain components of treatment (homework assignments, and so on), which could change the course of treatment. Such discussion between the consumer and provider may strengthen the therapeutic relationship.

Consumers are routinely surveyed to assess their satisfaction with services or their broader experiences with care (using, for example, measures such as the Consumer Assessment of Health Care Providers and Systems (CAHPS®) surveys and the CAHPS® Experience of Care and Health Outcomes survey for behavorial health care). Consumer-reported measures that attempt to assess the specific content of psychotherapy visits are less common. Miranda and colleagues (2010) have taken initial steps to develop a consumer-reported measure that assesses the use of evidence-based psychotherapy for depression. They developed survey items based on existing adherence and competence measures, on core therapy components identified in a review of literature, and in consultation with clinical experts in CBT, IPT, and psychodynamic therapy. They refined the measure using feedback from consumers who participated in cognitive interviews. They tested the measure in a sample of consumers who received services through a managed behavioral health care organization. Consumers were eligible to complete the survey if they had at least three outpatient psychotherapy visits for major depression in the prior six weeks. The consumers retrospectively rated the frequency with which their provider delivered each treatment element over the course of their treatment. The preliminary results suggest that the measure is reliable. The authors also reported that the consumer's reports on the provider's use of psychodynamic or cognitive behavioral approaches were associated with the provider's own identification with these therapeutic orientations, which the authors suggest provides preliminary evidence of the measure's validity. The study represents an important first step in developing a consumer-reported survey that assesses the fidelity of psychotherapy delivered to adults with depression. As suggested by the authors, the measure should be further tested in studies of CBT, IPT, and psychodynamic therapy.

2. Challenges Associated with Consumer-Reported Measures

There is some research to suggest that consumers may not be valid reporters for measures of the content delivered in psychotherapy. Two studies that examined reports from youth and their caregivers on their provider's delivery of mental health and substance abuse treatment found that they almost always reported that the provider delivered the therapeutic elements and that there was little agreement between their ratings and those by providers, clinical supervisors, and independent raters (Chapman et al. 2013; Schoenwald et al. 2009). Schoenwald et al. (2009) found that the consumer-reported ratings did not significantly vary over time. The researchers suggest that this could indicate that consumers may not be able to detect variation or changes in the delivery of specific psychotherapeutic content.

Consumers may have difficulty recalling therapy sessions. Respondents could complete surveys that collect data on the content of a single therapy visit, at multiple points in time, or retrospectively across multiple visits (Barber et al. 2007). The visits to be evaluated could be randomly selected or pre-determined. The greater the time lag between the therapy session and completion of the measure, the more difficult it may be for consumers to accurately report on the clinician's delivery of therapeutic elements.

Elements of psychotherapy change over the course of treatment. For example, providing information to the consumer about his or her problems or discussing expectations for treatment may be prescribed elements that occur at the beginning of therapy. In addition, the pace at which a provider delivers certain content may also vary from consumer to consumer, depending on the individual's rate of progress. Therefore, the timing and look-back period of a measure would have to account for these factors so providers or organizations are not inappropriately penalized for not delivering certain aspects of treatment or for not delivering them at the prescribed time. Asking the consumer to report the content of each visit directly after the visit might yield the most accurate assessment. Alternatively, a measure completed at a single point in time that attempts to capture therapy elements across several visits might address concerns about incorrectly penalizing a provider for not delivering certain elements at a certain time. Such an approach, however, would lend itself to recall problems.

Burden and costs of data collection. Introducing surveys to collect sufficient feedback from consumers to make reliable and fair comparisons of performance by individual providers or groups of providers would represent a new investment. Although on-site data collection might ease the cost of distribution, it could lead to bias (Anastario et al. 2010) (though bias might be able to be overcome as has been attempted in an on-site CAHPS® survey) (see the NQF endorsed CAHPS®Nursing Home Survey: Long-Stay Resident Instrument, NQF #0692). The use of electronic surveys could reduce the cost of data collection, especially if integrated into the EHR. However, newer methods of information collection (such as the use of patient portals or touch screen computers or tablets in the office) have not been well studied in psychotherapy.

3. Advantages Associated with Provider-Reported Measures

Provider self-reported measures of the content of psychotherapy have demonstrated promising results. Measures that ask providers to self-report the content of the psychotherapy they deliver have not been widely used outside of research settings or clinical supervision. Hepner and colleagues (2010), who worked with Miranda et al. (2010) to develop the consumer-reported measure of depression care described above, developed a parallel measure for providers. Providers were asked to select an individual currently in treatment for major depression and to rate the frequency with which they delivered each therapeutic element over the course of treatment. The results of testing the measure parallel the results for the consumer measure: there is support for the measure's reliability and preliminary evidence for its validity based upon positive associations between the reported use of psychodynamic or cognitive behavioral approaches and the providers' identification with these therapeutic orientations.

4. Challenges Associated with Provider-Reported Measures

The validity of provider-reported content of psychotherapy has not been fully established. Providers may over-estimate their delivery of specific treatment content, particularly if the measure is linked to performance appraisals or payment (Schoenwald et al. 2011b). Work by Martino and colleagues (2009) suggest that the validity of provider-reported measures may depend on the aspect of the psychotherapy being rated. They found that providers, supervisors, and external raters generally agreed on the provider's delivery of the basic elements of the particular treatment, such as the use of open-ended questions and affirmations; however, there was less agreement in their assessments of the providers' delivery of advanced psychotherapeutic treatment techniques. Providers rated their delivery of certain advanced components higher than did supervisors and external raters. Consistent with these findings, Chapman and colleagues (2013) found that providers over-estimated their ability to follow the treatment protocol when compared with independent raters.

Providers may have difficulty recalling therapy sessions. As mentioned previously, a measure of the delivery of psychotherapy could assess the care provided in a single therapy visit, at multiple points in time, or retrospectively across multiple visits. Just as with a consumer-reported version of the measure, the greater the recall period in completing the measure the more difficult it may be for providers to accurately recall the specific therapeutic elements delivered in a given session. As a result, providers may over-report the content of therapy delivered unless the information is recorded immediately following the session, perhaps as part of an EHR or some other electronic system.

5. Summary

Asking consumers or providers to report the content of psychotherapy may be burdensome. Although obtaining consumer perspectives on care is a well-established method for quality measurement, the validity of using consumer or provider surveys to report the content of psychotherapy for accountability purposes is unclear.

As mentioned above, an alternative to asking providers to report the content of visits through surveys or questionnaires would be to incorporate common fields in EHRs to capture this information. Asking providers about their usual treatment practices may be also more appropriate as part of a structure measure to assess competency.


View full report


"QualPsy.pdf" (pdf, 445.96Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®