The results of the agency analyses are reported by outcome domain in Tables 12-16. Overall, the results suggest that the quality ratings for most agencies and most outcomes are similar regardless of whether the current or alternative full model is used to risk-adjust outcomes. The difference tends to be minimal (no more than one to two percentage points) between the current and alternative risk-adjusted percent of an agencys patients with each outcome (see Figure 7). For a small share of agencies (i.e., those below the 5th or above the 95th percentile of the distribution), however, differences exceed four percentage points for Improvement in Ambulation, Improvement in Light Meal Preparation, Improvement in UTI, Acute Care Hospitalization, and Discharge to the Community (see columns 3 and 4 of Tables 12-16).
The average of the differences at each agency is greatest for Discharge to the Community (0.374 percentage points) followed by Improvement in UTI (0.287 percentage points). In the case of the UTI outcome, the average percent of patients improving at each agency was 83.7% when estimated using the current risk-adjustment model and 83.9% when estimated using the alternative full model. Despite the very small size of average differences, they often are statistically significant because sample sizes tend to be large, ranging from a low of 771 agencies when comparing the risk-adjusted Improvement in UTI outcomes, to 4,798 agencies in analyses of the percent of patients with an Acute Care Hospitalization.
While the magnitude of the difference between outcome estimates using the two risk-adjustment approaches is important, it is the ranking of each agency relative to others that is likely to be of most concern to providers. The next-to-the-last column in Tables 12-16 reports estimates of Spearmans rank correlation coefficient. These correlation coefficients are presented graphically in Figure 8. A value of one would indicate that rankings are exactly the same. For most outcomes, in fact, the correlation coefficient is close to one (i.e., it is above 0.950). The two lowest correlation coefficients are 0.912 for Improvement in UTI and 0.925 for Improvement in Ambulation.
The final column of each of the agency-level analysis tables reports the number and percent of agencies that change two or more deciles in rank when the risk-adjustment method is changed. (An agency, for example, would have to decline from the top decile--or top 10% in ranking--to the third decile or lower to be identified as changing two or more deciles.) The outcomes with the greatest number of agencies shifting at least two deciles in rank, not surprisingly, are those with the lowest Spearmans rank correlation coefficient. Among the agencies analyzed, 20.1% shifted two or more deciles in their Improvement in UTI ranking while 17.3% changed two or more deciles in their Improvement in Ambulation ranking.
Agency quality rankings differ the most where the difference in the explanatory power of the current and alternative risk-adjustment models is substantial. In the case of Improvement in Ambulation, the alternative risk-adjustment model explains considerably more of the variation in the outcome than the current model. It is the reverse for the Improvement in UTI outcome where the current model includes LOS among the risk-adjusters. Agency quality rankings for the utilization outcomes do not differ as much as might be expected given the exclusion of LOS from the alternative models and, as a result, the lower explanatory power of alternative versus current risk-adjustment models.
A sensitivity analysis then was conducted to better understand the impact on agency quality ratings of the inclusion of outcome-specific and OASIS prior items in the alternative risk-adjustment models of the OBQI quality indicators. Specifically, agency-level analyses were repeated with only the core risk-adjusters included in the alternative risk-adjustment models (i.e., the final version of Model 1 for each of the 31 currently risk-adjusted OBQI outcomes). The results of the sensitivity analysis are presented graphically in Figure 9 and Figure 10. The basic pattern of impacts is the same but, as expected, the difference in risk-adjusted outcomes using the current and alternative approaches increases (to between one and three percentage points for most agencies on almost all outcomes). For almost a third of the outcomes the Spearman rank correlation coefficient now is in the 0.900-0.950 range with the correlation coefficient for Improvement in Ambulation falling slightly below 0.900.
Finally, it is important to note that for many OBQI outcomes a relatively large number of agencies had fewer than 20 patients in the analytic sample with the potential to have the outcome. These agencies, therefore, were excluded when examining the impact of the alternative approaches to risk-adjustment on the percent of patients with the outcome. The number of agencies excluded is particularly large for two outcomes. All but 14.7% of agencies were excluded when examining the impact of alternative risk-adjustment approaches on estimates of Improvement in UTI and all but 19.5% were excluded when examining the impact on estimates of Improvement in Bowel Incontinence.