Alternative Risk-Adjustment Approaches to Assessing the Quality of Home Health Care: Final Report. Analytic Methods

07/01/2006

Analyses were conducted in two major phases (i.e., preliminary data analyses and final data analyses). Preliminary data analyses included replication of the CMS risk-adjustment models for the first set of 11 outcomes reported in HHQI and development of alternative models for these outcomes. A TAG meeting then was conducted with experts in home health care and risk-adjustment as well as policymakers and provider representatives. The TAG provided input on our initial approach based on the results of the preliminary data analyses. Following the TAG, a final set of alternative risk-adjustment models was developed for all 41 OBQI quality indicators and the impact of alternative risk-adjustment models on agency quality ratings was examined.

Logistic regression is the statistical method currently used to risk-adjust OBQI outcomes. We also used logistic regression when estimating risk-adjustment models since the purpose of the project was to replicate the existing approach and compare it with a theory and evidence-based approach to selecting risk-adjusters. An R-squared statistic and c statistic were estimated to assess the explanatory power and fit of current and alternative models.

The R-squared statistic is the squared correlation between the observed and predicted value of the dependent variable. This pseudo R-squared measure is the one estimated by the CMS contractor at the University of Colorado and included in publicly released reports describing current risk-adjustment models. While it is not equivalent to the R-squared statistic estimated in ordinary least squares regression, throughout this report we refer to increases and decreases in the R-squared statistics as changes in the “explanatory power” of a model. The change technically represents an increase or decrease in the extent of the agreement between observed and predicted values.

Preliminary Data Analyses

Preliminary analyses were conducted on the first set of OBQI outcomes publicly reported as part of HHQI.3 The 11 measures are:

  • Improvement in ambulation/locomotion;
  • Improvement in transferring;
  • Improvement in toileting;
  • Improvement in pain interfering with activity;
  • Improvement in bathing;
  • Improvement in management of oral medications;
  • Improvement in upper body dressing;
  • Improvement in confusion frequency;
  • Stabilization in bathing;
  • Admitted to an acute care hospital;
  • Any emergent care provided.

Current risk-adjustment models first were replicated to ensure that the samples for each model and specifications for independent and dependent variables in initial models exactly corresponded to those used by CMS when reporting the first set of HHQI outcomes. After replicating the risk-adjustment models for the 11 outcomes (a total of 15 models since three sub-models are estimated to risk-adjust Improvement in Transferring and Improvement in Pain Interfering with Activity) a theory and evidence-based approach was used to estimate alternative models for these outcomes.

Estimation of the theory and evidence-based models proceeded sequentially. A total of six models was estimated for each outcome. We began with a model limited to a core set of clinically relevant risk-adjusters, which included the baseline value of the outcome measure if it was not already among the core variables. We then added risk-adjusters at each subsequent step in the model building process.

  • Model 1: Clinical Core. Clinically relevant core variables plus the baseline value of the outcome measure if it is not among the core variables.

  • Model 2: Outcome Specific. Addition of other clinically relevant variables plausibly influencing the specific outcome except measures of health status prior to admission.

  • Model 3: OASIS “Prior” Items. Addition of prior health status variables (e.g., physical functioning 14 days prior to admission). The rationale for examining prior health status variables separately is because of questions regarding their reliability and possible elimination from the OASIS instrument.

  • Model 4: Clinical Therapies. Addition of indicators of whether the patient was receiving specific therapies at baseline (i.e., oxygen therapy, IV/infusion therapy, enteral/parenteral nutrition, and ventilator). The rationale for examining therapies separately from other clinically relevant risk-adjusters is that they are qualitatively different from the demographic and clinical characteristics of individuals. In addition, these therapies are used to determine the case-mix adjusted Medicare home health payment rate and might seem to be subject to home health agency “gaming.” Clinical and industry experts agree, however, that these services are invasive and would not be initiated without very clear clinical indications and medical orders.

  • Model 5: “Full Model” including Social Support. Addition of the living arrangement and social support indicators as risk-adjusters. We refer to this model as the “full model” since it includes all core variables available in the data set employed in the preliminary analyses, as well as risk-adjusters specific to the individual outcomes.

  • Model 6: Length of Stay (LOS). We added to the full models a home care episode LOS measure grouped into the categories employed by the University of Colorado. The sole purpose for including the LOS categories was to allow comparison of model statistics and parameter estimates with the University of Colorado risk-adjustment models.

The statistics below were estimated for the current and each of the alternative risk-adjustment models:

  • Number of OASIS items (i.e., the number of OASIS items that are the basis for the risk-adjusters included in the model).

  • Number of OASIS elements (some OASIS items include multiple elements with each element separately assessed and marked; e.g., M0290, “High Risk Factors,” for which smoking, obesity, alcohol, and drug dependency are all individual indicators--or elements--within the single OASIS item).

  • R-squared statistic (technically, a pseudo R-squared statistic that measures the extent of the agreement between observed and predicted values).

  • c statistic (a measure of how well the risk-adjusters in the model correctly classify the outcome examined; a completely inaccurate model would have a c statistic of 0.5, while a completely accurate model would have a c statistic of 1.0).

Technical Advisory Group Review of Preliminary Results

A one-day TAG meeting was convened with members, including industry representatives, having expertise in home health care quality, risk-adjustment, and home health care policy. The methodology and results of the preliminary analyses were summarized and provided to the TAG in a technical memorandum prior to the meeting. TAG members also received a technical memo reviewing the current CMS method for risk-adjusting OBQI outcome measures and other relevant literature on risk-adjustment of home health care outcomes. These documents served as the starting point for discussions at the TAG meeting.

The role of the TAG was to advise the project team on the development of the alternative risk-adjustment models, in particular, to provide advice on:

  • The selection of clinically and statistically sound variables from OASIS for the core set of risk factors;
  • The selection of risk-adjusters specific to an outcome indicator;
  • The sequential approach to model building employed in preliminary analyses;
  • OASIS items to eliminate as potential risk-adjusters.

Final Data Analyses: Risk-Adjustment Models

The analytic methods for estimating a final set of alternative risk-adjustment models were very similar to those used to estimate preliminary models. First, the remaining outcomes of the current risk-adjustment models were replicated. Following refinement of the core and supplementary risk-adjusters, three sequential models were estimated for all 31 home health quality indicators currently risk-adjusted in OBQI or HHQI.

  • Model 1: Clinical Core. Clinically relevant core variables plus the baseline value of the outcome measure if it is not among the core variables.

  • Model 2: Outcome Specific. Addition of other clinically relevant variables plausibly influencing the specific outcome except measures of health status prior to admission.

  • Model 3: OASIS “Prior” Items. Addition of prior health status variables (e.g., physical functioning 14 days prior to admission). The rationale for examining prior health status variables separately is because of questions regarding their reliability and possible elimination from the OASIS instrument.

The decision to estimate only three sequential models (as opposed to the six estimated in the preliminary analyses) was based on the advice of the TAG and further analysis of the social support risk-adjusters following the TAG meeting. The analysis confirmed that these factors contributed relatively little to the explanatory power of risk-adjustment models (see below).

Ten of the 41 OBQI quality indicators are not currently risk-adjusted. Only a model with the “clinical core” (i.e., Model 1) was estimated for each of these outcomes. The model statistics listed above in the Preliminary Data Analyses section were estimated for all risk-adjustment models developed in the Final Data Analyses.

Final Data Analyses: Agency Impacts

An agency-level analysis was conducted to examine how alternative approaches to risk-adjustment of the OBQI quality indicators affect an agency’s quality ratings. The agency-level analysis employed the validation data set provided by the University of Colorado with approximately 5,000 agencies included on the calendar year 2001 files. Two “adjusted” agency outcome rates were calculated for each of the 31 outcomes currently risk-adjusted in OBQI or HHQI. For example, an agency’s adjusted rate for Improvement in Bathing (see formula below) first was estimated using the current CMS risk-adjustment model. The adjusted rate then was re-estimated using the full alternative model developed to risk-adjust Improvement in Bathing in this project (i.e., the final version of Model 3). Not all agencies have estimates for all outcomes. If an agency has fewer than 20 patients with the potential to have an outcome, that outcome is not included in agency OBQI reports or in HHQI. We followed this approach and did not estimate the adjusted outcome for an agency when there were fewer than 20 patients with the potential to have the outcome.

There were five steps in the calculation of the adjusted agency outcome rate:

  1. Identify the patients at an agency with the potential to have an outcome.

  2. Determine the observed percent with the outcome at each agency where at least 20 patients have the potential to have the outcome.

  3. Estimate the predicted probability of the outcome at the individual level using: (1) the current risk-adjustment model, and (2) the final alternative model.

  4. Calculate the average predicted probability of the outcome at each agency when the current risk-adjustment model is used, and then when the alternative model is used.

  5. Adjust the agency mean so that agencies can be compared to the national average for an outcome using the formula published by the University of Colorado:

Adjusted Agency Outcome Rate = Observed Agency Outcome Rate +
(Observed National Outcome Rate - Agency Predicted Outcome Rate)

The following statistics then were estimated for each of the 31 outcomes:

  • Number and percent of agencies with the outcome (i.e., agencies with 20 or more episodes where the patient had the potential to have an outcome).

  • Mean and standard deviation of the absolute difference in the adjusted percent of patients at each agency with the outcome.

  • Percentage point difference at the 5th percentile of the distribution of differences in the adjusted percent of patients at each agency with the outcome.

  • Percentage point difference at the 95th percentile of the distribution of differences in the adjusted percent of patients at each agency with the outcome.

  • Rank of an agency based on the current risk-adjustment model (an integer number with 1 representing the best rank among all agencies).

  • Rank of an agency based on the alternative risk-adjustment model.

  • Percent of agencies with rankings that differ by two or more deciles (e.g., an agency is in the eighth decile using the current risk-adjustment method and in the sixth decile using the alternative model).

  • Simple t-test of the statistical significance of the absolute difference in the adjusted proportion of patients with the outcome.

  • Spearman’s rank correlation test of the association between the two rankings of agency performance as calculated using the current versus alternative risk-adjustment models.

A sensitivity analysis subsequently was conducted to better understand the impact on agency quality ratings of the inclusion of outcome-specific and OASIS “prior” items in the alternative risk-adjustment models of the OBQI quality indicators. Specifically, the agency-level analysis was repeated with only the core risk-adjusters included in the alternative risk-adjustment model for each of the 31 OBQI outcomes (i.e., the final version of Model 1). The results with and without the outcome-specific and OASIS “prior” items as risk-adjusters then were compared.


View full report

Preview
Download

"qualHH.pdf" (pdf, 3.72Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®