Final Report on the Effects of Sample Attrition on Estimates of Channeling's Impacts. D. Sensitivity Tests

01/13/1986

In addition to the heuristic approach of comparing impacts on Medicare-covered outcomes for full and analysis samples and the statistical approach of determining whether attrition bias exists, we also conducted some sensitivity tests as a way of assessing the effects of attrition. These results, presented in Wooldridge and Schore (forthcoming, Appendix E), show how estimates of channeling impacts on nursing home use would have changed if the full sample were available for analysis, under alternative assumptions about use of nursing homes by sample members not included in the nursing home sample.28 Three different procedures for imputing nursing home use to dropouts from the nursing home sample were employed:

  1. Overall mean usage levels for treatment and control groups were reestimated by forming a weighted average of mean use by sample members who survived the period (and had available data on nursing home use) and sample members who died within the period (but for whom data on use were available). The weights used were P and 1-P, respectively, where P was the proportion of the full sample that survived the entire period. This new estimate was intended to adjust for the underrepresentation in the nursing home sample of those who died within the analysis period, since it was felt that use by this group could be quite different from use by survivors.

  2. Estimates of mean use by nonrespondents were obtained that reflected observed differences between responders and nonresponders on screen characteristics.

  3. Estimates of mean use by nonrespondents were obtained that also reflected differences between respondents and nonrespondents on hospital use and Medicare-covered nursing home use during the analysis period for which total nursing home use was unknown.

The results of this analysis are reported in Wooldridge and Schore (forthcoming) and are simply summarized here. The alternative estimates were only slightly different from the original estimates obtained on the nursing home sample. The reasons for this are that: (1) contrary to expectations, those who die within the period generally had slightly fewer nursing home days on average than those who survived the period, for all four treatment/model groups, and (2) the observed characteristics (including prerandomization characteristics and concurrent hospital and nursing home use recorded in Medicare claims) of those in the nursing home sample do not differ greatly from the observed characteristics of those who were not in this sample. The average number of nursing home days for those who die within a period and those who survive the period are displayed below for the first six month period for the nursing home sample.

    Survivors     Decedents  
Treatment Group:
Basic Model 9.9 6.4
Financial Control Model   8.6 9.0
Control Group:
Basic Model 12.3 9.6
Financial Control Model 10.1 5.3

The similarity of use for decedents and survivors for both treatment and control groups suggests that the substantial underrepresentation of decedents in the nursing home sample does not lead to bias in the estimate means or impacts. Thus, the first alternative estimate must yield means and impact estimates that are not substantially different from those obtained previously. The similarity of most screen characteristics for persons included in and excluded from the nursing home sample resulted in imputed means for nursing home use for those excluded from the nursing home sample that were quite similar to those observed for the nursing home sample; hence, the second alternative yielded no substantive changes in estimates over the first alternative. Finally, those included in and those excluded from the nursing home sample were quite similar in their use of hospital and nurisng home days derived from Medicare claims; hence, estimates under the third alternative were relatively unchanged from those found using the other two approaches.

These results suggest that attrition, and most importantly, the consequent underrepresentation of persons who die within the analysis period, does not greatly distort estimates of channeling impacts on nursing home use. However, it may be the case that persons that were not included in the nursing home sample have very different nursing home use, even from those persons in the sample with similar characteristics. Even though the observed use of hospital and nursing home days derived from Medicare claims was similar for those included in and those excluded from the nursing home samples, this is not a guarantee that unobserved nursing home use that was paid for out of pocket or by Medicaid (the major payors) would be similar for those included in and those excluded from the nursing home samples. However, since actual nursing home values are unobserved for a portion of the sample, the approach of looking for and exploiting known differences between persons included in and excluded from the analysis sample is the only way to project what the use of those not in the sample actually was. It seems unlikely that those not in the sample would be so similar to those included on so many observed characteristics, some of which are known to affect or be correlated with nursing home use, and yet so different on unobserved characteristics that the results are seriously biased by the omission of these observations.


View full report

Preview
Download

"atritn.pdf" (pdf, 4.74Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®