HHS and other federal agencies support a wide range of technical assistance (TA) initiatives – including both virtual and in-person efforts – to support their significant investment in local communities working to improve economic resilience and well-being. Virtual TA is a form of planned and ongoing TA delivered remotely with built-in opport
Aligning Federal Performance Indicators Across Programs Promoting Self-Sufficiency: Key Considerations For Policymakers
This brief summarizes the current set of federal performance indicators and provides key policy considerations for policymakers and administrators within federal and state agencies who are interested in building a framework for coordinated performance measurement.
Aligning Federal Performance Indicators Across Programs Promoting Self-Sufficiency: Actionable Steps For Program Design And Practice
This brief outlines actionable steps that program designers at the federal, state, or local level can take to build or use aligned measures across programs in ways that can improve program management and increase service coordination.
CLIENT SURVEY OF THE DELIVERY OF EVIDENCE BASED PSYCHOTHERAPY Thank you for completing the Survey of the Delivery of Evidence Based Psychotherapy. Please read the following statement and choose "yes" or "no" below. CONSENT TO PARTICIPATE IN THE SURVEY OF THE DELIVERY OF EVIDENCE BASED PSYCHOTHERAPY I understand that:
SURVEY OF THE DELIVERY OF EVIDENCE BASED PSYCHOTHERAPY: SUPERVISORS Instructions: Please complete this survey based on your observation of the recent therapy sessions with ________________________________(clinician) and ________________________ (client). Note: Not every therapeutic element will be delivered in every therapy session.
SURVEY OF THE DELIVERY OF EVIDENCE BASED PSYCHOTHERAPY: CLINICIANS Instructions: Please complete this survey based on the recent therapy session you had with ______________________________ on _____________________. Note: Not every therapeutic element will be delivered in every therapy session.
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. I. Summary of Site Coordinator Debriefings
Site coordinators provided written feedback at the end of data collection. The following are topic areas of the types of feedback we received: Technological challenges . Some respondents had difficulty using the online survey links, whereas others found the links to be user-friendly. Both staff and consumer respondents at some sites found it ea
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. H. Stakeholder Feedback
In January 2015, we held four discussion groups with clinicians and supervisors, clients, site administrators, and health plans and payers to gather feedback on the measure's importance, face validity, usefulness, and feasibility. During this time, we also gathered feedback from site coordinators. Below, we summarize key themes identified across t
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. G. Results of Sensitivity and Specificity Analyses
To begin to understand the measure's validity, we calculated its sensitivity and specificity. For the purposes of this investigation, sensitivity is defined as the proportion of clinicians identified by clients or the clinicians themselves as high performers in the delivery of evidence-based psychotherapy when compared to supervisor scores. Specif
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. F. Approach to Creating a Measure Score
In order for a measure to be useful for performance and accountability purposes, the measure must discriminate performance and there must be a mechanism for scoring it to identify individuals who delivery evidence-based psychotherapy. As an initial approach to developing a measure score, we created standardized factor scores for each of the five f
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. E. Inter-Rater Agreement Results
Inter-rater reliability assesses the extent to which clinicians, supervisors, and clients agreed on whether the clinician delivered the survey element . We used the AC1 statistic, a measure of agreement adjusted for chance, to quantify agreement for the overall survey and at the item level.[ 7 ]
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. D. Internal Consistency Results
According to our KR20 analysis, the internal consistency of four out of five latent constructs is between 0.70 and 0.90 ( Table V.3 ; details shown in Appendix J ), which is in the "good" to "very good" range (Nunnally and Bernstein 1978). The internal consistency of Factor 4, suicide assessment, is between 0.54 and 0.69, which suggests some item
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. C. Confirmatory Factor Analysis
To further refine the scales identified in the EFA, we conducted CFAs on the five-factor model separately for the clinician, supervisor, and client samples.
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. B. Exploratory Factor Analysis
To identify the underlying factor structure of the survey, we fit a series of EFA models with varying numbers of latent factors (5, 6, 7, and 8). We examined the models' statistical fit and how well they corresponded to our theoretical understanding of the underlying constructs of evidence-based psychotherapy for PTSD.
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. A. Summary of Survey Administration
Survey mode . Eighty-nine percent of clinicians, 63 percent of supervisors, and 37 percent of clients completed the survey via the web ( Table V.1 ). The mode of survey completion varied by site. For example, in Site B, clients were provided the option of completing the survey immediately following the therapy session using the site's iPads. All
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. B. Stage 2--Pre-Testing the Measure
Once we finalized the development of the surveys, we collected quantitative and qualitative data to pre-test the measure. The quantitative data collection involved administering the surveys at specialty behavioral health organizations to assess the psychometric properties of the measure, potential approaches to scoring the measure, and potential i
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. A. Stage 1--Testing the Survey Items
After we developed the survey items, we then gathered qualitative information to answer the research questions in Table IV.1 . At this stage, the priority was to gather information regarding the face validity and interpretability of the survey items so we could refine them, as needed, in preparation for more formal measure testing in Stage 2. We
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. C. Survey Item Development
Once we finalized the common elements, we converted them into survey items (see Appendix E ). Three items drawn from another instrument were measured on a Likert scale; the remaining items consisted of categorical "yes or no" response options, with options for "don't know" or "don't remember." For example, to assess the element "challenging the c
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. B. Identification of Critical PTSD Psychotherapy Treatment Elements using an Established Methodology
Convening a TEP to generate a list of common treatment elements . As a first step in the development of this survey measure, we used the established "distillation and matching" approach (Chorpita 2005, 2009) to identify the elements present in evidence-based psychotherapy for adults with PTSD. Given the current research evidence, we focused on el
Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. A. Selection of Data Source
Based on feedback from the TAG and clinical and quality measure development experts, we determined that the information needed to calculate the measure was not available from claims or medical records and would therefore require primary data collection in the form of a survey. To reduce respondent burden, and based upon feedback from clinicians wi