Final Report on the Effects of Sample Attrition on Estimates of Channeling's Impacts. A. How Attrition Bias Occurs

01/13/1986

As indicated in the previous chapter, differences between treatment and control groups on screen characteristics that arise because of different patterns of attrition do not necessarily imply that estimates of channeling impacts are biased. If nonresponse was affected only by screen characteristics (e.g., ADL), then inclusion of these screen characteristics as explanatory (auxiliary control) variables in the outcome regression would control for the effects of attrition. This fact is not widely understood. Two conditions are necessary for impact estimates to be biased: (1) attrition is affected by the outcome measure being examined (e.g., whether in a nursing home), or by some unobserved factor (e.g., health status at the time of the attempted followup) that affects both attrition and the outcome measure, and (2) the pattern of attrition differs for treatment and control groups. Thus, finding attrition-induced differences between treatment and control groups on observed characteristics would have implied two things. First, impacts would have to be estimated by regression controlling for all initial characteristics on which treatments and controls differ as a consequence of attrition. Second, attrition-induced differences on observed characteristics would raise the suspicion that treatments and controls differ on unmeasured characteristics as well.

To see how different attrition patterns affect impact estimates, consider the following example. Suppose that in the full sample, 20 percent of the treatment group and 30 percent of the control group are in a nursing home at the 6-month followup. Thus, channeling reduced the probability of being in a nursing home by 10 percentage points. However, contrast the effects on these results under two different assumptions about the mechanism governing attrition. In the first case, assume that attrition was random within each experimental group--that is, attrition was affected only by experimental status: controls have a 70 percent probability of response and treatments have an 80 percent probability. Clearly, restricting the analysis to just the responders would have no effect on the impact estimate: since all treatment group members have the same probability of response, the proportion of responders in nursing homes is the same as the proportion for the full sample (20 percent), and similarly 30 percent of the controls who respond to the interview will be in nursing homes. Hence, the impact estimate of 10 percentage points is unaffected by attrition, even though the attrition rates are different for treatments and controls. Furthermore, although we do not show this here, the same results hold if attrition is affected by any screen characteristic that is controlled for.

Consider how our conclusions about the effects of attrition change, however, if the probability of response is also affected by the value of the outcome being examined. For example, suppose that the probability of response among those who are in a nursing home at followup (Y=I) is only 50 percent for treatment group members and 40 percent for control group members. Suppose that the probabilities of response for those not in nursing homes (Y=0) are 95 percent for the treatment group and 85 percent for the control group (i.e., for each value of Y, treatment group members are 10 percentage points more likely to respond than controls). In this case, the treatment and control group means for the responders only (R=1) are:15

Treatment group:   Estimated proportion in nursing homes for responders    =    responders in nursing homes
(responders in nursing homes + responders not in nursing homes)
   =    0.20 * 0.50 * N / (0.20 * 0.50 + 0.80 * 0.95) * N
   =    11.6%
 
Control group: Estimated proportion in nursing homes    =    0.30 * 0.40 * N / (0.30 * 0.40 + 0.70 * 0.84) * N
   =    16.8%

Note that the proportion of responders in nursing homes is much smaller than the values that would have been obtained if no attrition occurred (about half as large). More important, however, we see that subtracting the control group mean from the treatment group mean for respondents gives a predicted impact of channeling on nursing home placement of -5.2 percentage points, about half the true impact in this example.

These overly simplified examples demonstrate how different mechanisms of attrition may or may not cause bias in impact estimates. Overall attrition rates for treatment and control groups in the second example are 14 and 28.5 percent, respectively--quite close to the rates actually observed for some of our analysis samples. An attrition mechanism of this type could result in estimated impacts that are too small to be statistically significant, for cases in which the estimate for the full sample would have been large and highly significant.

The statistical correction procedure described in this chapter controls for attrition by determining whether responding sample members who have a relatively low predicted probability of remaining in the sample (given their screen characteristics) are more likely to have a larger (or smaller) than expected value of the outcome (Y), given the values of auxiliary control variables. This is determined by obtaining for each observation a predicted "attrition-correction" term and including it in the regression equation used to estimate channeling impacts. If this type of correlation between unobserved factors in the attrition equation and unobserved factors in the outcome equation does exist: the coefficient on the correction term will have a significant coefficient. If, in addition, attrition is affected by experimental status, the estimated channeling impact will change substantially. The statistical procedure is described below.

View full report

Preview
Download

"atritn.pdf" (pdf, 4.74Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®