Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. IV. APPROACH TO MEASURE TESTING


Following the specification of the measure, we pre-tested it in two stages. In the first stage, we used qualitative methods to gather information on the importance and usefulness of the measure, the validity of the survey items, and the feasibility of using this type of measure.

In the second stage, we used both quantitative and qualitative methods. Stage 2 of the pre-testing effort was designed to gather information to inform potential uses of the measure, assess preliminary information on its psychometric properties, explore approaches to developing a measure score, and gather information on the measure's implementation.

Although it is formative in nature, our testing initiative was designed to lay a foundation for additional measure testing, and if, appropriate, for future endorsement by the NQF. As such, we framed many of the research questions around the NQF measure criteria importance, scientific acceptability, usability, and feasibility, defined by the NQF as:

  • Importance. The strength of evidence supporting that a measure concept promotes high quality care and allows for differentiation in performance.

  • Scientific acceptability. The verification that the psychometric properties of a measure--validity and reliability--are strong enough to justify its use to assess quality of care:

    • Validity. The ability of measure specifications to promote accuracy in data collection and measure score calculation to ensure appropriate characterization of performance.

    • Reliability. The ability of measure specifications to promote consistency in data collection and aggregation to ensure that variability in measure score reflects actual variation in performance.

  • Usability. The value of a measure in informing quality improvement activities.

  • Feasibility. The availability of data elements required for the calculation of a measure, whether a measure is susceptible to inaccuracies, and the level of effort involved in collecting and calculating the measure.

This chapter describes the methods used to test each of these criteria. We briefly summarize the testing questions and specific methods for each stage of measure testing.