After we developed the survey items, we then gathered qualitative information to answer the research questions in Table IV.1. At this stage, the priority was to gather information regarding the face validity and interpretability of the survey items so we could refine them, as needed, in preparation for more formal measure testing in Stage 2. We also gathered preliminary stakeholder input on the measure's importance, usability, and feasibility. Below, we describe the process we used to gather qualitative information, and then summarize the information learned.
|TABLE IV.1. Qualitative Research Questions|
|Importance||Is the measure appropriate for assessing quality of care?|
|Validity||Are stakeholders interpreting the survey items in the way we intended them to be interpreted?
Are there alternate ways of wording key concepts that better resonate with stakeholders?
Do the survey items measure quality of psychotherapy?
|Usability||How would different organization and entities use this measure to improve the quality of PTSD care?|
|Feasibility||How burdensome is the measure to complete? Can the measure be accurately scored?|
1. Telephone Discussions
From May 2013 to October 2013, we hosted a series of telephone discussions with stakeholders to gather input on the wording and interpretation of the survey items, the usefulness of the measure to improving quality of PTSD care, and the feasibility of completing the measure. We utilized an iterative process to gathering information whereby we held a discussion with stakeholders, revised the survey items based upon the feedback received, and then held additional discussions with new stakeholders. Discussion group participants represented two types of stakeholder groups:
Clinicians and clinical supervisors. Participants included nine clinicians and clinical supervisors who were experienced in providing psychotherapy to adults with PTSD. They were identified through recommendations by project team members and members of the TEP. We hosted a total of four discussions with clinicians and supervisors, with group sizes ranging from one to three participants.
Clients. Participants included six adults (three men and three women) who were either nearing the end of treatment for PTSD or had completed treatment within the previous year. They included both military veterans and civilians and were identified through recommendations from TEP members, ASPE, project team members, and postings on listservs such as the National Alliance for the Mentally Ill's listserv. We hosted three discussions with clients, with group sizes ranging from one to two participants. Participants received a $20 gift card.
We drafted specific questions to fit the particular expertise of each type of discussion group and revised the questions for each subsequent discussion.
2. Summary of Stakeholder Input
Survey items. Participants in both stakeholder groups provided valuable suggestions on how to improve the clarity and meaning of the survey items. Based upon their input, we altered the wording of some items, added concrete examples to further clarify the items, and combined redundant items.
Importance and usefulness. There was general agreement on the importance of improving the quality of PTSD care; however, not all participants saw value in this specific measure. Some clinicians argued that the measure only assesses the delivery of CBT when other therapies are also effective in treating PTSD; as a result, the measure would not be useful or of interest to clinicians who do not provide CBT. Others, particularly clinical supervisors, felt it would be useful. Some clients indicated the importance of improved outcomes rather than the therapeutic techniques used by their clinicians. Other clients appreciated the effort to improve the quality of their care and saw value in the measure.
Feasibility. Stakeholder feedback on the feasibility of using the measure varied. Some clinicians and supervisors expressed concern regarding the length of the measure, the time required to complete it, and the feasibility of completing it within 24 hours of a given therapy session. Clients did not indicate concerns with the measure's length.
3. Development of the Final Survey
Once we completed revisions to the survey items based on feedback obtained from the telephone discussions, we provided TEP members with the opportunity to review and comment on a revised version of the survey. ASPE and NIMH also conducted a final review. We then made minor revisions and finalized the survey items, which consisted of 32 items in the clinician and supervisor versions and 25 items in the client version. The final surveys are available in Appendix E.
Once the survey items were finalized, we created web-based surveys using Opinio software (Version 6.7.1; ObjectPlanet, Inc., Norway). Prior to launching data collection, we rigorously tested the program to ensure that response fields functioned properly; users could move back and forth between questions, change answers, and save and return to the survey to complete at a later time; and entered responses were correctly coded and stored.