Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. VI. CONCLUSIONS AND NEXT STEPS

05/01/2019

The development of a measure of the delivery of evidence-based psychotherapy has the potential to improve the quality of care for adults with PTSD. We made promising strides in creating the foundation of such a measure; however, a significant amount of additional work is needed to develop a final measure that can be used for accountability purposes. Below, we provide overarching conclusions and recommended next steps.

  • Additional input. Although there was support for use of the measure in training and education, support for using it for accountability purposes was limited. Additional input from a larger group of stakeholders regarding the measure's use for internal quality improvement and the circumstances under which it would be useful would inform the next stages of measure development.

  • Further revisions. Our analyses suggest that the survey assesses important underlying constructs associated with the delivery of evidence-based treatment for PTSD and that many survey items produce significant agreement across the three raters. The analyses also suggest that several items need refinement. For example, items with low inter-rater agreement and/or low internal consistencies may be candidates for deletion. Items with significant cross-loadings and moderate agreement could need revision. The surveys should be revised further, with additional cognitive testing and stakeholder input conducted on the refinements.

  • Further investigation of feasibility. Several stakeholders expressed concern regarding the measure's feasibility. Refinement to the survey items may result in a shorter measure that takes less time to complete, which should improve the feasibility of using it. In addition, it would be useful to have additional information from a larger group of stakeholders regarding topics such as preferred survey mode (including mobile technology applications), the available infrastructure to support the measure, and approaches to automating aspects of site coordination.

  • Further development of the measure for broader application. The factor analyses results identified therapeutic constructs that are likely relevant in the delivery of psychotherapy for conditions other than PTSD. The measure could be refined and further tested to create modules that broadly apply to the delivery of psychotherapy.

  • Examine inter-rater reliability and factor structure with revised items and larger sample. Once the survey items have been refined, additional work will be needed to test whether the refinements improve inter-rater agreement and the factor structure. The goal of our current project was to pre-test this instrument. A pilot test with a larger sample offering increased diversity in sites, clinicians, and clients would increase the external validity of the measure.

  • Examine other scoring methods. Our current thresholds for high and low delivery of evidence-based psychotherapy yielded positive results in terms of specificity and sensitivity. After item refinement, these scoring methods should be verified and compared to other possible methods of scoring. For example, contextual scoring may be beneficial, as it would allow clinicians flexibility in deviating from a treatment plan for appropriate reasons (for example, in cases where a clinician did not use an expected set of therapeutic elements, because he or she had to help a client manage suicidal ideation).

  • Additional validity testing. Additional psychometrics are needed to validate this measure. The use of an external, independent rater (not associated with the site) to serve as the preferred gold standard is important. To assess the measure's predictive validity, information on patient outcomes (for example, symptom improvement, quality of life, and functioning) is critical.

The measure developed under this project has the potential to address significant gaps in quality of PTSD care. Additional work is needed to further prepare it for implementation on a larger-scale basis and to better understand the groups and situations where the measure will be most useful.