Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. H. Stakeholder Feedback


In January 2015, we held four discussion groups with clinicians and supervisors, clients, site administrators, and health plans and payers to gather feedback on the measure's importance, face validity, usefulness, and feasibility. During this time, we also gathered feedback from site coordinators. Below, we summarize key themes identified across the discussions. Given overlapping themes in the feedback provided, we include information learned from the site coordinator briefings in this section.

Importance. Stakeholders agreed on the importance of improving the quality of PTSD care. Perceptions regarding this measure's importance varied. Health plans indicated a strong preference for outcomes measures and indicated that additional process measures have little utility in improving quality of care.

Validity. Perceptions regarding the measure's face validity were mixed.

  • Measuring true quality of treatment. Several clinicians, administrators, and health plan/payer representatives, and site coordinators suggested the measure was too narrowly focused on cognitive behavioral approaches and did not cover the range of (perceived appropriate) treatments for adults with PTSD. Others felt the survey items reflected the true quality of evidence-based treatment.

Usability. Stakeholders had mixed opinions regarding the usefulness of the measure.

  • Usefulness. Stakeholders agreed the measure would be useful for training and continuing education purposes; however, there was a lack of consensus regarding its usefulness for quality improvement. Clients, administrators, and some clinicians suggested the measure would also be useful for accountability and quality improvement; however, health plan/payer representatives uniformly agreed the measure would not be of use. Given the relatively small proportion of their beneficiaries who are in treatment for PTSD and the emphasis on the development of outcome measures, the health plan and payer group representatives would not find the measure useful.

  • Service setting. Stakeholders suggested the measure could be useful for outpatient clinics, the VA, day hospital programs, and PTSD Centers of Excellence. Site administrators and health plan/payer representatives did not perceive the measure as being useful for health plans.

  • Unintended consequences. Some clients suggested that survey completion might unintentionally result in a potential confrontation between clients and clinicians. Participants in the client group offered a scenario in which a client indicated that his clinician did not provide most of the items on the survey. In this scenario, participants worried that the client's survey responses would be shared with the clinician and influence the nature of the subsequent session. In order to avoid this potential scenario, some clients suggested making the survey anonymous to the clinician. In contrast, others from the client group said that they would the opportunity to influence their course of treatment. This group of clients stated that if their clinician were not receptive to the feedback, they would discontinue treatment and find a new clinician. Some clients with good relationships with their clinicians indicated this was an unlikely scenario.

  • Other concerns. Administrators stated that recording or directly observing therapy sessions could hinder clients' willingness to complete the survey or make them wary of speaking freely during a session out of fear of repercussions. Some clients also expressed concerns about unintended consequences and specifically about how clients might react if, based on the survey, they felt the clinician was not delivering quality care.

Feasibility. With the exception of clients, all stakeholder groups expressed concerns regarding the measure's feasibility.

  • Prioritization of surveys. All stakeholder groups suggested it would be too resource intensive to utilize all three versions of the survey. Given the time, resources, and (in some cases) changes in supervision processes that would be required, none of the groups selected the supervisor version of the survey for administration. Health plan/payer representatives indicated a preference for the client version. Site administrators and clinicians indicated they would choose either the clinician or the client version.

  • Survey length. Health plan/payer representatives and some site coordinators felt the survey was too long.

  • Survey mode. Stakeholder feedback on the feasibility of implementing web-based surveys varied. Some stakeholders found it convenient and time-saving; others experienced challenges in navigating the online survey and indicated that many clients do not have reliable Internet access.

  • Coordination. Administrators and site coordinators expressed concerns regarding the feasibility of coordinating the data collection effort, particularly in drawing the sample and providing reminders to the participants to complete the survey. The administrator from one site also indicated concerns regarding the resources required to translate the materials into other languages.