Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. V. TESTING RESULTS

This chapter summarizes the quantitative and qualitative results of the measure testing. We first present summary statistics on survey administration. We then summarize the factor analyses, internal consistency, inter-rater reliability, measure performance, and sensitivity and specificity. We conclude the chapter with stakeholder feedback.

• A. Summary of Survey Administration

  • Survey mode. Eighty-nine percent of clinicians, 63 percent of supervisors, and 37 percent of clients completed the survey via the web (Table V.1). The mode of survey completion varied by site. For example, in Site B, clients were provided the option of completing the survey immediately following the therapy session using the site's iPads. All of the clients at this site completed the survey using the web; 63 percent of the clients opted to complete it before leaving the site following their session (data not shown). Conversely, 100 percent of clients at Site D elected to complete the survey on paper.

    Length of time to complete the survey. On average, clinicians completed the web survey in 8 minutes and supervisors and clients in 10 minutes (Table V.1).[6] We excluded from these calculations 17 cases where the response times were greater than one hour. It is likely that these outlying values reflect individuals who started the survey, saved their responses, and completed the survey at a later time.

    Length of time between therapy session and survey completion. To reduce recall bias, clients and clinicians were asked to complete the survey within 24 hours of the therapy session, and supervisors were asked to complete it within 24 hours of their review of the session. Table V.1 suggests that, on average, clinicians and clients did not complete the survey within this 24-hour window. The average number of days between when the therapy session occurred and when clinicians and clients completed the survey was 9.6 days (range 0-127 days) and 2.0 days (range 0-12 days), respectively. We do not have information on when the supervisors began their review of the therapy session; however, the average length of time between the occurrences of the therapy session and when supervisors completed the survey was 20 days (range 0-102 days).

    Multiple factors may contribute to the length of time between the occurrence of the therapy session and survey completion. Conversations with site coordinators indicate that in some cases the length of time may be an artifact of clinicians and supervisors saving their survey responses but not actually clicking the "submit" button to transmit them. If the survey were to undergo future testing, revisions to the web version could provide additional prompts to submit the survey upon completion. Additionally, some site coordinators indicated that supervisors conducted weekly supervision and reviewed session tapes in batches; this may contribute in part to the delayed completion of the surveys. It is also likely that the data may accurately reflect the time needed for clinicians and supervisors to complete the survey, in which case, further investigation is needed into recall bias and the accuracy of the data when the survey is completed days and sometimes weeks after the therapy session occurred. Further investigation may also be needed into the organizations' capacity to complete this type of quality measure, and into the resources -- and perhaps changes in internal processes -- needed to facilitate more timely survey completion. In considering processes that facilitate data collection, regular reminders to staff to complete the survey appear key. The coordinators at Sites C and E were especially responsive to Mathematica alerts to remind staff of outstanding surveys, and these sites have comparatively shorter survey completion times. Routine reminders to clinicians and supervisors to complete the measure may be an important part of collecting the data in a timely way.

    TABLE V.1. Summary of Survey Administration: Modes and Completion Times
    		Number of Completed Surveys	Percentage Web-Based Complete (n)	Percentage Paper-Based Complete (n)	Average Number of Minutes to Complete the Survey (range)*,**	Average Number of Days from Therapy Session Start Date to Survey Completion (range)*,***
    Total	Clinicians	96	89% (85)	11% (11)	8 (2-56)	9.9 (0-127)
    	Supervisors	97	63% (76)	37% (21)	10 (2-52)	19.6 (0-102)
    	Clients	78	37% (29)	63% (49)	10 (3-30)	1.9 (0-12)
    Site A	Clinicians	34	100% (34)	0% (0)	9 (3-56)	14 (0-127)
    	Supervisors	34	100% (34)	0% (0)	10 (2-47)	27 (0-76)
    	Clients	22	23% (5)	77% (17)	6 (3-11)	3.8 (0-7)
    Site B	Clinicians	10	100% (10)	0% (0)	9 (2-13)	20 (0-72)
    	Supervisors	10	100% (10)	0% (0)	7 (3-20)	24 (0-102)
    	Clients	8	100% (8)	0% (0)	7 (4-11)	2.5 (0-9)
    Site C	Clinicians	15	100% (15)	0% (0)	6 (2-13)	0.5 (0-4)
    	Supervisors	15	100% (15)	0% (0)	9 (4-52)	8 (2-20)
    	Clients	14	7% (1)	93% (13)	15 (15)	0 (0)
    Site D	Clinicians	12	58% (7)	42% (5)	10 (2-23)	6.6 (0-14)
    	Supervisors	14	100% (15)	0% (0)	12 (4-23)	13.6 (0-51)
    	Clients	13	0% (0)	100% (13)	NA	NA
    Site E	Clinicians	18	89% (16)	11% (2)	6 (3-14)	4.4 (0-29)
    	Supervisors	17	18% (3)	82% (14)	9 (9)	2.7 (0-7)
    	Clients	17	82% (14)	18% (3)	11 (6-30)	1 (0-12)
    Site F	Clinicians	7	43% (3)	57% (4)	8 (6-9)	8.3 (1-22)
    	Supervisors	7	0% (0)	100% (7)	NA	NA
    	Clients	4	25% (1)	75% (3)	26 (26)	1 (1)
    * Paper-based completes are excluded, because the information is not available. ** Durations over one hour were excluded (17 cases out of 191 total), as it is likely that these participants completed the survey in more than one sitting. *** Days calculated are calendar days.

    Item-Level Missing Information. Most participants entered a response for each survey item. On the clinician survey, eight items had missing information and the missingness ranged from 0-2 percent (Appendix G, Table G.1). On the supervisor survey, 28 items had missing information; the level of missing information ranged from 0-3 percent (Appendix G, Table G.2). On the client survey, 30 items had missing information, which ranged from 0-6 percent (Appendix G, Table G.3).

• B. Exploratory Factor Analysis

  • To identify the underlying factor structure of the survey, we fit a series of EFA models with varying numbers of latent factors (5, 6, 7, and 8). We examined the models' statistical fit and how well they corresponded to our theoretical understanding of the underlying constructs of evidence-based psychotherapy for PTSD.

    According to the model fit statistics (Appendix H, Table H.1), all four of the EFA models represented the underlying data structure very well, suggesting that from a statistical standpoint any of these models could inform the CFA. We then examined the factor structures for parsimony and clinical meaning. The five-factor model provided the most parsimonious solution with the least number of significant cross-loadings. This solution was also the most interpretable based on constructs identified during the measure development stage. For these reasons, we retained the five-factor model (see Table V.2) for further validation at the CFA stage.

    In grouping items into factors, we considered items with factor scores of 0.40 or above. If an item cross-loaded on multiple factors, we assigned it to the factor where it had the highest loading and/or for which other factors related to the item also scored highly. Below, we describe the factor groupings and the labels we assigned to each factor.

    • Factor 1: Structuring and conducting the therapy session. Ten items compose Factor 1 and include aspects of treatment such as creating an agenda, setting treatment goals with the client, soliciting client feedback on treatment, and being directive.

    • Factor 2: Psychoeducation and therapeutic techniques. Fifteen items make up Factor 2. The majority of items are therapeutic techniques (that is, cognitive restructuring, Socratic method, imagining the traumatic event) and psychoeducation (providing education about symptoms and/or the traumatic event).

    • Factor 3: Therapeutic alliance. Three items from the therapeutic alliance measure make up Factor 3.

    • Factor 4: Assessment. Two items on assessment loaded on this factor.

    • Factor 5: Homework. All six of the items that loaded on this factor are related to assigning, reviewing, and encouraging homework completion.

    Five items were not statistically significantly associated with any of the factors. These items included the suicide risk assessment questions, use of Socratic questioning, the facilitation of alternate hypotheses, and one question on the clinician's time management.

    TABLE V.2. The Five-Factor EFA Solution
    Clinician Survey Item Number	Factor 1	Factor 2	Factor 3	Factor 4	Factor 5
    12.a. IMAGINE		0.699*
    12.b. WRITE		0.783*
    12.c. OTHER SOCRAT	-0.266*	0.867*
    12.d. REAL		0.578*
    26. TRUST			0.750*
    24. CONFIDENT			0.774*
    25. LIKES			0.818*
    1. AGENDA	0.790*
    2. REVIEW AGENDA	0.719*	0.376*		-0.290*
    3. BACKGROUND	0.307*	0.676*
    4. EXPECTATIONS	0.696*
    5. GOALS	0.647*
    10. IDENTIFY		0.670*
    7. COG RESTRUC		0.469*
    8. SOCRAT
    9. FACILITATE
    10. OTHER IDENTIFY		0.556*
    11. TECHNIQUES		0.633*
    13. DISCUSS	0.226*	0.549*
    14. STRUGGLE
    15. DIRECTIVE	0.853*
    16. TX FEEDBACK	0.604*	0.233*
    17. TH FEEDBACK	0.431*
    18. ASSIGN					0.769*
    19. REVIEW INSTRUC					0.722*
    20. ADDRESS					0.949*
    21. SOLUTION					0.749*
    22. REVIEW HMWK					0.925*
    23. ENCOURAGE					0.874*
    27.a. EVER SUIC
    27.b. TODAY SUIC	0.269*
    28.a. EVER USE SUIC
    28.b. TODAY USE SUIC
    29.a. EVER INSTRU				0.614*
    29.b. TODAY INSTRU	0.466*			0.750*
    30.a. EVER SYMP EDU		0.699*		-0.393*
    30.b. TODAY SYMP EDU		0.836*
    31.a. EVER TRAUMA ED		0.574*
    31.b. TODAY TRAUMA ED		0.768*			-0.254*
    32.a. EVER OUTLINE	0.516*
    32.b. TODAY OUTLINE	0.685*	0.369*
    12. OVERALL TECHNIQUES		0.737*
    * Factor loadings not significant at p < 0.05 were excluded from the table to facilitate interpretation of the results.

• C. Confirmatory Factor Analysis

  • To further refine the scales identified in the EFA, we conducted CFAs on the five-factor model separately for the clinician, supervisor, and client samples. The CFA models fit the data well and had a similar factor structure across the different respondents (Appendix H, Table H.2), suggesting that the instrument may function similarly across the three types of respondents. Detailed CFA results by respondent type are available in Appendix I. A summary of the commonalities and differences in the factor structures across the samples is below:

    • Factor 1: Structuring and conducting the therapy session. The number of items that compose Factor 1 varies by respondent type. Across the three samples, five items related to agenda setting, goals, treatment process and expectations, and treatment feedback make up this factor. In the clinician and supervisor surveys, this factor also comprises reviewing agendas and being directive. Outlining the treatment process and symptom assessment also loaded on Factor 1 in the clinician and client surveys.

    • Factor 2: Psychoeducation and therapeutic techniques. The items that compose Factor 2 are nearly identical across the three samples and, as previously described, focus on therapeutic techniques such as the use of Socratic questioning and cognitive restructuring.

    • Factor 3: Therapeutic alliance. The three therapeutic items compose Factor 3 across all three samples.

    • Factor 4: Assessment. This factor has only one item, suicide risk assessment "today," shared between the three samples. Each paired sample (clinician/client, clinician/supervisor, client/supervisor) has common items that make up this factor. The items include therapeutic techniques and additional assessment questions.

    • Factor 5: Homework. The items that compose Factor 5 are nearly identical. It has four common items across the three samples and five common items between the clinicians and supervisors.

    Summary. Taken together, the EFA and CFA results suggest that the survey items measure constructs relate to the delivery of psychotherapy for PTSD. For further instrument development, we recommend analyzing whether core items that are consistent across all three samples are sufficient to capture the corresponding latent factors without sacrificing the reliability of a scale. This could help to shorten the measurement instrument and decrease the burden for respondents while retaining essential measurement properties. We also recommend considering modifications to the fourth factor, which only has one item shared by all three samples and which also has the lowest scale reliability of all five factors.

• D. Internal Consistency Results

  • According to our KR20 analysis, the internal consistency of four out of five latent constructs is between 0.70 and 0.90 (Table V.3; details shown in Appendix J), which is in the "good" to "very good" range (Nunnally and Bernstein 1978). The internal consistency of Factor 4, suicide assessment, is between 0.54 and 0.69, which suggests some items may need revision. On average, we observed the highest reliability across all domains in the supervisor sample, followed by the clinician and client samples.

    TABLE V.3. Internal Consistency Results by Factor and Respondent
    Respondent	Factor 1	Factor 2	Factor 3	Factor 4	Factor 5
    Clinician	0.78	0.83	0.82	0.58	0.81
    Supervisor	0.88	0.89	0.85	0.69	0.81
    Client	0.77	0.77	0.82	0.54	0.90

• E. Inter-Rater Agreement Results

  • Inter-rater reliability assesses the extent to which clinicians, supervisors, and clients agreed on whether the clinician delivered the survey element. We used the AC1 statistic, a measure of agreement adjusted for chance, to quantify agreement for the overall survey and at the item level.[7]

    Inter-rater agreement between clinicians, supervisors, and clients. All three raters completed the survey on 76 therapy sessions and at least two raters completed it on 97 therapy sessions. The weighted agreement for the whole survey ranged from 0.39 to 0.58 across different rater pairs (Table V.4), which is considered fair-to-moderate agreement (Gwet 2014). Supervisors and clinicians had the highest weighted inter-rater agreement; supervisors and clients and clinicians and clients had comparable inter-rater agreement.

    TABLE V.4. Inter-Rater Reliability for Clinicians, Supervisors, and Clients
    Raters	AC1	SE	CI	Significance Level
    Supervisor-clinician-client	0.43	0.005	(0.34-0.53)	<0.01
    Supervisor-clinician	0.58	0.04	(0.51-0.65)	<0.01
    Supervisor-client	0.39	0.07	(0.25-0.54)	<0.01
    Client-clinician	0.39	0.07	(0.26-0.51)	<0.01
    NOTE: AC1 values above 0.80 suggest high agreement; 0.61-0.80 substantial agreement, 0.41-0.60 moderate agreement, 0.21-0.40 fair agreement, and 0-0.20 slight agreement.

    In addition to calculating inter-rater agreement for the whole measure, we also calculated it at the item level. Across the three raters, item percentage agreement ranged from 39 percent to 90 percent and the AC1 statistic ranged from -0.09 to 0.86 (Table V.5). Items for which there was only slight agreement included two homework-related items, one therapeutic technique item, and one item on managing the therapy session. Similar trends occurred when examining item-level agreement between each rater pair (clinicians/supervisors, clinicians/clients, supervisors/clients) with high agreement in ratings of some survey items and low agreement in others (see Appendix K).

    TABLE V.5. Item-Level Inter-Rater Agreement Between Clinicians, Supervisors, and Clients
    	Overserved Agreement	AC1	CI	Significance Level
    Did you and your therapist discuss an agenda or plan for your session?	85.48%	0.42	<0.01	(0.24-0.60)
    Did your therapist talk about or check-in on your expectations of how therapy will go?	62.50%	0.38	<0.01	(0.20-0.56)
    Did your therapist work with you to set goals you both agreed on?	65.45%	0.52	<0.01	(0.35-0.69)
    Did your therapist help you become aware of or realize feelings, views or thoughts in your life that have been influenced by your traumatic experience? These might include feelings, views, or thoughts about being safe in the world, the presence of danger, trust, and self-esteem.	76.71%	0.69	<0.01	(0.57-0.82)
    Did your therapist ask you several direct questions to make you think critically about or examine your thoughts, feelings, or beliefs? For example, your therapist might ask: How do you know this? Can you give me an example? What are some other ways of viewing this? What are the pros and cons to your way of thinking about this? How did you come to this conclusion? What evidence do you have to justify this?	58.62%	0.48	<0.01	(0.31-0.64)
    Did your therapist offer other ways of thinking about your issues (e.g., problem areas or areas you want to work on) related to the trauma? For example: Thought: "I can't trust anyone." Thought suggested by therapist: "Some people can't be trusted, but there are other people who are trustworthy."	64.81%	0.41	<0.01	(0.23-0.60)
    Did you and your therapist discuss people, events, or places you now avoid or stay away from because of your traumatic experience? For example, someone in a car accident might avoid driving on the freeway.	61.67%	0.20	<0.01	(0.01-0.39)
    Did your therapist do any of the following things to help you deal with fear, anxiety or things you now avoid because of your trauma? Ask you to imagine or retell your traumatic experience for longer than 10 minutes. Ask you to write about your traumatic experience. Ask you questions to make you think critically about or examine your thoughts, feelings, or beliefs related to your fear, anxiety, and avoidance of things (i.e., "How do you know this? Can you give me an example?"). Ask you to do real world experiments like visiting a place related to the traumatic experience for longer than 10 minutes.	49.30%	0.22	<0.01	(0.06-0.38)
    After you described your traumatic experience, did you and your therapist discuss the details of what happened to you, how it impacted your life, or your emotions about the event?	62.86%	0.19	<0.01	(0.03-0.35)
    Did your therapist make good use of your session time today?	76.19%	-0.09	<0.01	(-0.23-0.05)
    Did your therapist ask for your opinion on how your treatment is going?	66.07%	0.50	<0.01	(0.34-0.66)
    Did your therapist ask for feedback on how she/he is doing in helping you recover from your PTSD?	45.10%	0.25	<0.01	(0.07-0.44)
    Did your therapist assign homework or practice assignments (to be completed by the next session) to work on your PTSD symptoms or problem areas?	58.82%	0.33	<0.01	(0.16-0.50)
    Did your therapist make sure you understood how to complete your homework for the next session?	65.52%	0.45	<0.01	(0.29-0.61)
    If you had problems completing your previously assigned homework, did your therapist work with you to come up with solutions to these problems?	66.07%	0.09	<0.01	(-0.08-0.27)
    Did your therapist review and discuss your homework from the previous session?	54.55%	0.17	<0.01	(-0.01-0.35)
    When reviewing the homework from the previous session, did your therapist encourage or provide you with constructive feedback?	60.00%	0.22	<0.01	(0.03-0.40)
    My therapist and I have built mutual trust.	10.67%	0.85	<0.01	(0.80-0.90)
    I am confident in my therapist's ability to help me.	9.33%	0.76	<0.01	(0.70-0.83)
    I believe my therapist likes me as a person.	17.33%	0.86	<0.01	(0.81-0.91)
    Has your therapist ever asked you if have had thoughts about committing suicide?	90.00%	0.79	<0.01	(0.68-0.89)
    During this session, did your therapist ask you if you had thoughts about committing suicide?	39.39%	0.61	<0.01	(0.47-0.76)
    Has your therapist ever asked you to answer questions about your PTSD symptoms? This might include completing a form before or after therapy.	78.69%	0.28	<0.01	(0.12-0.43)
    During this session, did your therapist ask you about your PTSD symptoms? This might include completing a form or survey before or after therapy.	61.54%	0.44	<0.01	(0.29-0.59)
    Has your therapist ever provided information about PTSD and PTSD symptoms?	84.75%	0.86	<0.01	(0.77-0.95)
    During this session, did your therapist provide information about PTSD and PTSD symptoms?	60.94%	0.26	<0.01	(0.09-0.43)
    Has your therapist ever provided with specific education on the nature of the traumatic event (i.e., facts about the type of trauma)? For example, this might include education on the nature of sexual assault, or how sexual assault generally influences your viewpoints and beliefs.	68.00%	0.48	<0.01	(0.31-0.65)
    During this session, did your therapist ever provide you with specific education on the nature of the traumatic event (i.e., facts about the type of trauma)? For example, this might include education on the nature of sexual assault, or how sexual assault generally influences your view points and beliefs	56.67%	0.22	<0.01	(0.06-0.38)
    Has your therapist ever explained how your particular treatment will work?	78.18%	0.77	<0.01	(0.65-0.89)
    During this session, did your therapist explain how your particular treatment will work?	67.21%	0.21	<0.01	(0.05-0.36)

    Implications for survey revisions. Although there was high agreement between raters for several survey items, the inter-rater agreement results suggest that several items may benefit from further investigation and potential revision. Examples of items with low agreement and/or poor AC1 values include:

    • Two questions regarding Socratic discussion methods.

    • Two questions about therapeutic techniques to deal with avoidance.

    • One question about emotional reprocessing regarding the emotions surrounding the traumatic event.

    • One question regarding the psychoeducation about the nature of the traumatic event.

    It is possible that these and other items with low agreement could be revised by further simplifying the questions or providing more detailed examples; however, further cognitive interviewing may be needed to better understand how stakeholders interpret them. Alternatively, the items may need to be deleted.

• F. Approach to Creating a Measure Score

  • In order for a measure to be useful for performance and accountability purposes, the measure must discriminate performance and there must be a mechanism for scoring it to identify individuals who delivery evidence-based psychotherapy. As an initial approach to developing a measure score, we created standardized factor scores for each of the five factors identified in the factor analyses. The scores were standardized to have a mean of zero and a standard deviation of one. A total standardized score was also created using the same method. As depicted in Figure V.1, the distribution in total scores varies for each of the three respondent types.

    FIGURE V.1. Distribution of Total Standardized Score by Respondent Type

    Next we examined approaches to establishing measure thresholds that could be used to identify clinicians who deliver evidence-based psychotherapy. We examined four thresholds: the mean, median, mean plus one standard deviation, and the 75th percentile. We selected two thresholds -- the median and 75th percentile -- as more conservative and liberal estimates of measure performance for further investigation. In the subsequent section, we describe the measure's performance when using these thresholds.

• G. Results of Sensitivity and Specificity Analyses

  • To begin to understand the measure's validity, we calculated its sensitivity and specificity. For the purposes of this investigation, sensitivity is defined as the proportion of clinicians identified by clients or the clinicians themselves as high performers in the delivery of evidence-based psychotherapy when compared to supervisor scores. Specificity, in contrast, is the proportion of clinicians identified as low performers in the delivery of evidence-based psychotherapy. We compared clinician and client scores to the supervisor scores, which for the purposes of these analyses, we treated as the gold standard. We examined the implications for the measure's sensitivity and specificity using two thresholds, the median (P5) and above the 75th percentile (P75) to determine high and low delivery of evidence-based psychotherapy.

    Table V.6 summarizes the sensitivity and specificity results. For supervisors and clinicians, the sensitivity rate ranged from 0.32 to 0.78 across the factors. The specificity rate ranged from 0.51 to 0.88. For supervisors and clients, the sensitivity rate was 0.22-0.61 and the specificity rate was 0.49-0.81 (Table V.6).

    Based on these preliminary findings, the P50 (median) threshold appears to better discriminate performance than the more stringent P75 threshold. This threshold obtained consistently higher values for sensitivity and specificity in supervisor-clinician pairings when compared to the P75 threshold.

    In both supervisor-clinician and supervisor-client pairings, the P75 threshold demonstrated higher specificity. However, in supervisor-client pairings, the sensitivity values with the P75 were quite low compared to those observed among the clinicians at the same threshold, suggesting a differential performance with the instrument between respondents. The observed differences in performance across pairings suggest a need to further evaluate the instrument to identify the optimal threshold for each respondent type.

    When thinking about measure implementation, it is important to note there may be instances where a supervisor is not the gold standard. For example, supervisors may treat too few patients to serve as experts in the delivery of evidence-based psychotherapy or they may not be trained in cognitive behavioral approaches--which the measure largely draws upon--and therefore, may not be best positioned to identify a clinician's use of these techniques. In Chapter VI, we discuss next steps for further assessing the measure's validity.

    TABLE V.6. Results of Sensitivity and Specificity Analyses
    	Comparison of Supervisor and Clinician Scores				Comparison of Supervisor and Client Scores
    	Specificity P50 Threshold	Sensitivity P50 Threshold	Specificity P75 Threshold	Sensitivity P75 Threshold	Specificity P50 Threshold	Sensitivity P50 Threshold	Specificity P75 Threshold	Sensitivity P75 Threshold
    Factor 1: Structuring and conducting the Session	0.78	0.78	0.81	0.45	0.49	0.50	0.81	0.32
    Factor 2: Psychoeducation and therapeutic techniques	0.51	0.50	0.78	0.32	0.56	0.56	0.76	0.26
    Factor 3: Therapeutic Alliance	0.63	0.64	0.82	0.48	0.50	0.51	0.75	0.22
    Factor 4: Suicide assessment	0.61	0.63	0.85	0.57	0.56	0.58	0.79	0.37
    Factor 5: Homework	0.63	0.63	0.80	0.42	0.62	0.61	0.76	0.32
    Overall score	0.73	0.74	0.85	0.50	0.61	0.62	0.76	0.32

• H. Stakeholder Feedback

  • In January 2015, we held four discussion groups with clinicians and supervisors, clients, site administrators, and health plans and payers to gather feedback on the measure's importance, face validity, usefulness, and feasibility. During this time, we also gathered feedback from site coordinators. Below, we summarize key themes identified across the discussions. Given overlapping themes in the feedback provided, we include information learned from the site coordinator briefings in this section.

    Importance. Stakeholders agreed on the importance of improving the quality of PTSD care. Perceptions regarding this measure's importance varied. Health plans indicated a strong preference for outcomes measures and indicated that additional process measures have little utility in improving quality of care.

    Validity. Perceptions regarding the measure's face validity were mixed.

    • Measuring true quality of treatment. Several clinicians, administrators, and health plan/payer representatives, and site coordinators suggested the measure was too narrowly focused on cognitive behavioral approaches and did not cover the range of (perceived appropriate) treatments for adults with PTSD. Others felt the survey items reflected the true quality of evidence-based treatment.

    Usability. Stakeholders had mixed opinions regarding the usefulness of the measure.

    • Usefulness. Stakeholders agreed the measure would be useful for training and continuing education purposes; however, there was a lack of consensus regarding its usefulness for quality improvement. Clients, administrators, and some clinicians suggested the measure would also be useful for accountability and quality improvement; however, health plan/payer representatives uniformly agreed the measure would not be of use. Given the relatively small proportion of their beneficiaries who are in treatment for PTSD and the emphasis on the development of outcome measures, the health plan and payer group representatives would not find the measure useful.

    • Service setting. Stakeholders suggested the measure could be useful for outpatient clinics, the VA, day hospital programs, and PTSD Centers of Excellence. Site administrators and health plan/payer representatives did not perceive the measure as being useful for health plans.

    • Unintended consequences. Some clients suggested that survey completion might unintentionally result in a potential confrontation between clients and clinicians. Participants in the client group offered a scenario in which a client indicated that his clinician did not provide most of the items on the survey. In this scenario, participants worried that the client's survey responses would be shared with the clinician and influence the nature of the subsequent session. In order to avoid this potential scenario, some clients suggested making the survey anonymous to the clinician. In contrast, others from the client group said that they would the opportunity to influence their course of treatment. This group of clients stated that if their clinician were not receptive to the feedback, they would discontinue treatment and find a new clinician. Some clients with good relationships with their clinicians indicated this was an unlikely scenario.

    • Other concerns. Administrators stated that recording or directly observing therapy sessions could hinder clients' willingness to complete the survey or make them wary of speaking freely during a session out of fear of repercussions. Some clients also expressed concerns about unintended consequences and specifically about how clients might react if, based on the survey, they felt the clinician was not delivering quality care.

    Feasibility. With the exception of clients, all stakeholder groups expressed concerns regarding the measure's feasibility.

    • Prioritization of surveys. All stakeholder groups suggested it would be too resource intensive to utilize all three versions of the survey. Given the time, resources, and (in some cases) changes in supervision processes that would be required, none of the groups selected the supervisor version of the survey for administration. Health plan/payer representatives indicated a preference for the client version. Site administrators and clinicians indicated they would choose either the clinician or the client version.

    • Survey length. Health plan/payer representatives and some site coordinators felt the survey was too long.

    • Survey mode. Stakeholder feedback on the feasibility of implementing web-based surveys varied. Some stakeholders found it convenient and time-saving; others experienced challenges in navigating the online survey and indicated that many clients do not have reliable Internet access.

    • Coordination. Administrators and site coordinators expressed concerns regarding the feasibility of coordinating the data collection effort, particularly in drawing the sample and providing reminders to the participants to complete the survey. The administrator from one site also indicated concerns regarding the resources required to translate the materials into other languages.

• I. Summary of Site Coordinator Debriefings

  • Site coordinators provided written feedback at the end of data collection. The following are topic areas of the types of feedback we received:

    Technological challenges. Some respondents had difficulty using the online survey links, whereas others found the links to be user-friendly. Both staff and consumer respondents at some sites found it easier to complete paper copies of the surveys.

    Survey questions. Some sites found the questions to be too targeted to CPT and prolonged exposure therapy, which could skew the results, since not all clients sampled received that type of treatment. Some site coordinators heard from supervisors and clinicians that the survey questions were a useful reminder to stick to evidence-based treatment and to utilize certain tactics in all sessions. Some clinicians found the questions to be too generic.

    Participant hesitance:

    • Clients. Many sites struggled with client hesitance about participating in the study. Some were initially wary of having an observer present during their session or having their session audio taped, but coordinators said that most clients forgot about the observer or audio tape by the end of the session. However, sites reported that, overall, many clients were excited to participate and, despite initial hesitance, were willing to participate if it could help others receive high quality care in the future.

    • Clinicians. Many sites reported that most of the clinicians were cooperative and excited to participate. However, some were hesitant about being observed and some were unclear about what would happen with the results of the survey.

    Scheduling and time commitment:

    • Many sites reported that tracking client appointments and client absences and re-schedules was challenging.

    • One site did not fully understand the supervisor time commitment (to observe or review every selected session in its entirety) when they originally agreed to participate.

    • Some sites did not fully understand the site coordinator time commitment, and found that the role was too much for one person. One site mentioned that internal logistics were challenging.

    • One site would have liked more time for data collection.