Final Report on the Effects of Sample Attrition on Estimates of Channeling's Impacts. VI. SUMMARY AND CONCLUSIONS


The experimental design of the channeling evaluation was chosen to ensure that the experience of the control group would provide a reliable estimate of what would have occurred to treatment group members in the absence of the demonstration. However, attrition from this carefully drawn sample can thwart these intensions if the sample available for analysis is not comparable for the two groups. Regression models were used in the evaluation to control for observable differences between the treatment and control groups that could arise because of attrition, but estimates may still be biased if the two groups differ on unobservable characteristics. This will occur if (1) those sample members for whom data are available differ on unobservable characteristics from those for whom data are not available, (2) those unobservable factors also affect outcomes of interest, and (3) rates or patterns of attrition differ for treatment and control groups.

For each of the major areas of analysis in the evaluation, an analysis sample was defined which included those observations in the research sample for which the data necessary for analysis were available. Thus, the following analysis samples were defined:

  • 6/12 and 18 month Medicare samples (for hospital outcomes)
  • 6, 12, and 18 month nursing home samples (nursing home outcomes)
  • 6, 12, and 18 month followup samples (well-being outcomes)
  • 6, 12, and 18 month in-community samples (formal and informal care outcomes)

The percent of the full sample included in most of these analysis samples was substantially greater for treatments than for controls, especially in the financial control model. Thus, one of the conditions that could lead to bias was present. These differences are shown to be due primarily to treatment/control differences in response rates at the baseline interview. However, despite this difference in rates of attrition, the analysis samples exhibited only minor treatment/control differences on initial screen characteristics.

To investigate whether impact estimates based on these analysis samples were likely to be biased because of attrition, two types of analyses were performed. First, Medicare data, which were available for virtually the entire research sample, were used to construct several variables measuring the amount of Medicare-covered services used, including hospital days and expenditures, nursing home days and expenditures, and several types of formal community-based and physician services. We then estimated channeling impacts on these Medicare-only variables using the full sample, and again on the various analysis samples. These two sets of estimates were then compared to determine whether limiting the analysis to those in the analysis samples produced different estimates than would have been obtained for the full sample.

We found that for the variables examined, the impact estimates obtained on the analysis samples rarely differed substantively from those for the full sample. This was especially true for the Medicare sample. Since over 98 percent of all hospital use by sample members was covered by Medicare, it was clear that attrition led to no bias in estimated impacts on hospital outcomes. For other outcomes and samples the results were less clear cut. Although there were few instances of noteworthy differences between the full and analysis samples on the Medicare data, these data covered only a fraction of the total use of nursing homes and formal services and contained no data at all on other key outcomes, including well-being and informal care. Thus, we could not be certain that estimated impacts on these other outcomes would not be biased by attrition. Alternative procedures were required to determine whether attrition bias for these outcomes was present.

A statistical model designed to control for the nonrandom selection of an analysis sample was used for this purpose. For each analysis sample, a model was estimated to predict which of the full sample observations were retained in the analysis, as a function of personal characteristics measured on the screening interview. Each estimated "sample inclusion" model was then used to construct an additional variable for each member of the corresponding analysis sample, which, when included as an additional control variable in the regression equation used to estimate channeling impacts, controls for the effects of attrition. The coefficient on the constructed attrition bias term was then tested for statistical significance to deter-mine whether the condition necessary for regression estimates to be biased by sample attrition was met.

In general, this procedure yielded very little evidence of attrition bias. The estimated correlations between unobserved factors affecting attrition and those affecting a given outcome variable were typically small and rarely significantly different from zero. Impact estimates obtained from the regressions which included the control variable for the effects of attrition were very similar to the impact estimates obtained without this correction term.

Finally, to ensure that the results obtained from the statistical correction procedure were not distorted by overly restrictive assumptions, we developed a somewhat more general model that would take into account two possible differences between treatments and controls and between models: differences in the relationship between observed (screen) characteristics and attrition, and differences in the covariance between unobserved factors affecting attrition and those affecting the outcome variable under examination. Use of this more general procedure showed (1) that the attrition models were not very different for treatments and controls or for basic and financial control models, and (2) that although there were some substantive differences between the 4 treatment/model groups in the correlations between unobserved factors, controlling for them separately yielded no convincing evidence that the unadjusted estimates were biased by attrition.

Although both the heuristic and statistical approaches led us ultimately to conclude that attrition bias was not a major problem, there were a number of isolated results that, if viewed alone, would have caused greater concern about attrition. To further ensure that no important evidence of attrition bias was being overlooked, the results from the Medicare data analysis were compared to those obtained from the statistical approaches for each outcome area to see if the alternative approaches both indicated that attrition bias might be a problem for any given set of outcomes. The specific patterns of attrition implied by the two procedures were also compared for consistency.

Estimates of impacts on hospital outcomes were shown conclusively to be unaffected by attrition, based on Medicare data alone. For nursing home outcomes, the Medicare comparison showed no evidence of bias in the estimates, and the only evidence to the contrary from the statistical procedure was two cases in which impact estimates changed in statistical significance. However, in both of these instances, the impact estimates changed only marginally after controlling for the effects of attrition going from slightly below the critical value for statistical significance to slightly above it (and vice versa). Furthermore, the results that ostensibly controlled for the effects of attrition had the implausible implication that the bias was in one direction at 6 months and in the opposite direction at 12 months, and occurred only in the basic model. Finally, the sensitivity tests performed showed no evidence of bias. Thus, it seems clear that estimates of impacts on nursing home outcomes were not biased by attrition.

For well-being outcomes, the Medicare data provide no direct evidence concerning attrition bias, but comparing the full and followup sample estimates of impacts on a few of the Medicare-covered services examined suggested that bias was potentially a problem only for the basic model, and only at six months. However, the results from the statistical procedure to measure attrition bias implied that there was no bias in any of the well-being outcome measures examined in any time period for either model.

For formal and informal care outcomes, estimates of impacts on service use from the in-community sample are very similar to those obtained on the full sample in all three time periods for the financial control model, and at 12 and 18 months in the basic model. However, at 6 months in the basic model, estimated impacts on skilled nursing visits and reimbursements were statistically significant for the analysis sample but not for the full sample. This suggests that the in-community sample estimates of impacts on use of formal care (and possibly informal care) may be overstated in this time period for the model because of attrition. However, the impact estimates on the two samples do not differ in statistical significance for most outcomes even in this period, nor is the magnitude of the difference that great even for skilled nursing (13 percent of the control group mean for the full sample estimate compared to about 24 percent of the control group mean for the analysis sample estimate). The lack of evidence of bias at 12 months and in the other model led us to doubt further that attrition bias is a major problem for formal and informal care impact estimates. For formal care outcomes, this conclusion is further supported by the results from the statistical analyses, which indicate an absence of the conditions necessary for attrition bias and strong similarity between impact estimates obtained using the procedure to control for the possible effects of attrition and estimates obtained without such control.

For informal care outcomes the evidence is less clear cut. The results from the initial statistical procedure showed no evidence of bias, but the other, less restrictive statistical approach of controlling for attrition effects led to results that implied serious bias in the estimates for both models. Whereas the unadjusted results implied no effect of channeling on informal care in the basic model, and (at most) modest reductions in the financial control model, the latter adjusted estimates showed large, statistically significant reductions in informal care in the basic model and no reductions in the financial control. Also, both the Medicare and more general statistical approaches implied similar patterns of attrition, i.e., that the systematic attrition occurred mainly for the treatment group in the basic model. However, a number of factors were cited that suggest that this result was a statistical anomaly rather than credible evidence of severe attrition bias. Hence, we concluded that informal care impact estimates were probably not biased by attrition either.

The two approaches used in this report each have their flaws. The heuristic approach of seeing how estimated impacts on some variables change when the analysis is restricted to a subset of the full sample is appealing because it is a direct measure of attrition bias, albeit for variables other than those in which we are most interested. Reliance on these results as proof that there is no attrition bias in the estimated impacts on outcomes that we are interested in requires belief that any unobserved factors affecting both attrition and the outcome of interest also affect the Medicare outcomes. Although this assumption may be plausible, it obviously cannot be verified.

The statistical approach is also appealing, but for different reasons--it pertains to precisely the outcome variables of interest, provides a direct test of whether there is bias in the estimates obtained on the analysis sample, and also offers a way to obtain unbiased estimates of impacts on any outcome. The more general model developed and used here adds to the attractiveness of this approach by making the results sensitive to potentially different observed and unobserved patterns of attrition for treatment and control groups. However, in either statistical model the estimates may be quite sensitive to the assumptions of the model (bivariate normal Disturbance terms in the outcome and sample inclusion equations ), may reflect other nonlinear relationships between the outcome and control variables that have nothing to do with attrition, and are sensitive to colinearity between the correction term and the control variables in the outcome equations.

Despite these flaws, the two approaches employed here greatly exceed what is normally done or is possible to do to examine attrition bias, because the data available from the screen and Medicare/FCS claims on nonrespondents greatly exceeds what is usually available on sample dropouts. By definition, it is never possible to know with certainty what results would have been obtained had no sample attrition occurred. The heuristic and statistical approaches are the best methods available to assess the effects of attrition on our impact estimates, and both approaches provide convincing evidence that the inferences drawn from the analysis samples about the existence and magnitude of channeling impacts are no different from what would be drawn if the full sample were available for analysis.

View full report


"atritn.pdf" (pdf, 4.74Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®